Anyone tinkering with Python long enough has been bitten (or torn to pieces) by the following issue:
def foo(a=[]):
a.append(5)
return a
Python novices would expect this function called with no parameter to always return a list with only one element: [5]
. The result is instead very different, and very astonishing (for a novice):
>>> foo()
[5]
>>> foo()
[5, 5]
>>> foo()
[5, 5, 5]
>>> foo()
[5, 5, 5, 5]
>>> foo()
A manager of mine once had his first encounter with this feature, and called it "a dramatic design flaw" of the language. I replied that the behavior had an underlying explanation, and it is indeed very puzzling and unexpected if you don't understand the internals. However, I was not able to answer (to myself) the following question: what is the reason for binding the default argument at function definition, and not at function execution? I doubt the experienced behavior has a practical use (who really used static variables in C, without breeding bugs?)
Edit:
Baczek made an interesting example [1]. Together with most of your comments and Utaal's in particular [2], I elaborated further:
def a():
print("a executed")
return []
def b(x=a()):
x.append(5)
print(x)
a executed
>>> b()
[5]
>>> b()
[5, 5]
To me, it seems that the design decision was relative to where to put the scope of parameters: inside the function, or "together" with it?
Doing the binding inside the function would mean that x
is effectively bound to the specified default when the function is called, not defined, something that would present a deep flaw: the def
line would be "hybrid" in the sense that part of the binding (of the function object) would happen at definition, and part (assignment of default parameters) at function invocation time.
The actual behavior is more consistent: everything of that line gets evaluated when that line is executed, meaning at function definition.
Actually, this is not a design flaw, and it is not because of internals or performance. It comes simply from the fact that functions in Python are first-class objects, and not only a piece of code.
As soon as you think of it this way, then it completely makes sense: a function is an object being evaluated on its definition; default parameters are kind of "member data" and therefore their state may change from one call to the other - exactly as in any other object.
In any case, the effbot (Fredrik Lundh) has a very nice explanation of the reasons for this behavior in Default Parameter Values in Python [1]. I found it very clear, and I really suggest reading it for a better knowledge of how function objects work.
[1] https://web.archive.org/web/20200221224620id_/http://effbot.org/zone/default-values.htma
would "resurface" instead of being recreated every call? - Torxed
x = [ [1,2,3] ] * 5
is also a seriou source of bugs, but not a design flaw. - Elazar
var numbersIterator = function () {var i=0; return function () {return i++;}}()
, will probably give you nightmares. - Mark Amery
functions are objects
. In your paradigm, the proposal would be to implement functions' default values as properties rather than attributes. - bukzor
sum([[1], [2]], [])
will result in empty list always being an empty list? What's the difference? - Hi-Angel
Suppose you have the following code
fruits = ("apples", "bananas", "loganberries")
def eat(food=fruits):
...
When I see the declaration of eat, the least astonishing thing is to think that if the first parameter is not given, that it will be equal to the tuple ("apples", "bananas", "loganberries")
However, suppose later on in the code, I do something like
def some_random_function():
global fruits
fruits = ("blueberries", "mangos")
then if default parameters were bound at function execution rather than function declaration, I would be astonished (in a very bad way) to discover that fruits had been changed. This would be more astonishing IMO than discovering that your foo
function above was mutating the list.
The real problem lies with mutable variables, and all languages have this problem to some extent. Here's a question: suppose in Java I have the following code:
StringBuffer s = new StringBuffer("Hello World!");
Map<StringBuffer,Integer> counts = new HashMap<StringBuffer,Integer>();
counts.put(s, 5);
s.append("!!!!");
System.out.println( counts.get(s) ); // does this work?
Now, does my map use the value of the StringBuffer
key when it was placed into the map, or does it store the key by reference? Either way, someone is astonished; either the person who tried to get the object out of the Map
using a value identical to the one they put it in with, or the person who can't seem to retrieve their object even though the key they're using is literally the same object that was used to put it into the map (this is actually why Python doesn't allow its mutable built-in data types to be used as dictionary keys).
Your example is a good one of a case where Python newcomers will be surprised and bitten. But I'd argue that if we "fixed" this, then that would only create a different situation where they'd be bitten instead, and that one would be even less intuitive. Moreover, this is always the case when dealing with mutable variables; you always run into cases where someone could intuitively expect one or the opposite behavior depending on what code they're writing.
I personally like Python's current approach: default function arguments are evaluated when the function is defined and that object is always the default. I suppose they could special-case using an empty list, but that kind of special casing would cause even more astonishment, not to mention be backwards incompatible.
("blueberries", "mangos")
. - Ben Blank
some_random_function()
appends to fruits
instead of assigning to it, the behaviour of eat()
will change. So much for the current wonderful design. If you use a default argument that's referenced elsewhere and then modify the reference from outside the function, you are asking for trouble. The real WTF is when people define a fresh default argument (a list literal or a call to a constructor), and still get bit. - alexis
fruits
is intentionally a tuple
here to prevent the case you describe. tuple
s are immutable. It cannot be appended. That choice also clearly suggests that the author does not expect it to change. - jpmc26
global
and reassigned the tuple - there is absolutely nothing surprising if eat
works differently after that. - user3467349
globals x; x = new_value
, I'll become very cautious and kinda expect weird behaviour somewhere else. Python is dynamic, it's easy to break it if you want. It shouldn't shoot itself in the foot automatically, though. - Eric Duminil
The relevant part of the documentation [1]:
[1] http://docs.python.org/reference/compound_stmts.html#function-definitionsDefault parameter values are evaluated from left to right when the function definition is executed. This means that the expression is evaluated once, when the function is defined, and that the same “pre-computed” value is used for each call. This is especially important to understand when a default parameter is a mutable object, such as a list or a dictionary: if the function modifies the object (e.g. by appending an item to a list), the default value is in effect modified. This is generally not what was intended. A way around this is to use
None
as the default, and explicitly test for it in the body of the function, e.g.:def whats_on_the_telly(penguin=None): if penguin is None: penguin = [] penguin.append("property of the zoo") return penguin
function.data = []
) or better yet, make an object. - bukzor
I know nothing about the Python interpreter inner workings (and I'm not an expert in compilers and interpreters either) so don't blame me if I propose anything unsensible or impossible.
Provided that python objects are mutable I think that this should be taken into account when designing the default arguments stuff. When you instantiate a list:
a = []
you expect to get a new list referenced by a
.
Why should the a=[]
in
def x(a=[]):
instantiate a new list on function definition and not on invocation? It's just like you're asking "if the user doesn't provide the argument then instantiate a new list and use it as if it was produced by the caller". I think this is ambiguous instead:
def x(a=datetime.datetime.now()):
user, do you want a
to default to the datetime corresponding to when you're defining or executing x
?
In this case, as in the previous one, I'll keep the same behaviour as if the default argument "assignment" was the first instruction of the function (datetime.now()
called on function invocation).
On the other hand, if the user wanted the definition-time mapping he could write:
b = datetime.datetime.now()
def x(a=b):
I know, I know: that's a closure. Alternatively Python might provide a keyword to force definition-time binding:
def x(static a=b):
class {}
block to be interpreted as belonging to the instances :) But when classes are first-class objects, obviously the natural thing is for their contents (in memory) to reflect their contents (in code). - Karl Knechtel
static
keyword in a python function signature doesn't make sense. If the signature is evaluated once (how python does it), then the default is also only evaluated once, and thus "static" is redundant. If the signature is not evaluated once (how python doesn't do it), then "static" would be evaluated every time the function is called - which wouldn't be static. - MisterMiyagi
Well, the reason is quite simply that bindings are done when code is executed, and the function definition is executed, well... when the functions is defined.
Compare this:
class BananaBunch:
bananas = []
def addBanana(self, banana):
self.bananas.append(banana)
This code suffers from the exact same unexpected happenstance. bananas is a class attribute, and hence, when you add things to it, it's added to all instances of that class. The reason is exactly the same.
It's just "How It Works", and making it work differently in the function case would probably be complicated, and in the class case likely impossible, or at least slow down object instantiation a lot, as you would have to keep the class code around and execute it when objects are created.
Yes, it is unexpected. But once the penny drops, it fits in perfectly with how Python works in general. In fact, it's a good teaching aid, and once you understand why this happens, you'll grok python much better.
That said it should feature prominently in any good Python tutorial. Because as you mention, everyone runs into this problem sooner or later.
property
s -- which are actually class level functions that act like normal attributes but save the attribute in the instance instead of the class (by using self.attribute = value
as Lennart said). - Ethan Furman
I'm really surprised no one has performed the insightful introspection offered by Python (2
and 3
apply) on callables.
Given a simple little function func
defined as:
>>> def func(a = []):
... a.append(5)
When Python encounters it, the first thing it will do is compile it in order to create a code
object for this function. While this compilation step is done, Python evaluates* and then stores the default arguments (an empty list []
here) in the function object itself. As the top answer mentioned: the list a
can now be considered a member of the function func
.
So, let's do some introspection, a before and after to examine how the list gets expanded inside the function object. I'm using Python 3.x
for this, for Python 2 the same applies (use __defaults__
or func_defaults
in Python 2; yes, two names for the same thing).
>>> def func(a = []):
... a.append(5)
...
After Python executes this definition it will take any default parameters specified (a = []
here) and
cram them in the
__defaults__
attribute for the function object
[1] (relevant section: Callables):
>>> func.__defaults__
([],)
O.k, so an empty list as the single entry in __defaults__
, just as expected.
Let's now execute this function:
>>> func()
Now, let's see those __defaults__
again:
>>> func.__defaults__
([5],)
Astonished? The value inside the object changes! Consecutive calls to the function will now simply append to that embedded list
object:
>>> func(); func(); func()
>>> func.__defaults__
([5, 5, 5, 5],)
So, there you have it, the reason why this 'flaw' happens, is because default arguments are part of the function object. There's nothing weird going on here, it's all just a bit surprising.
The common solution to combat this is to use None
as the default and then initialize in the function body:
def func(a = None):
# or: a = [] if a is None else a
if a is None:
a = []
Since the function body is executed anew each time, you always get a fresh new empty list if no argument was passed for a
.
To further verify that the list in __defaults__
is the same as that used in the function func
you can just change your function to return the id
of the list a
used inside the function body. Then, compare it to the list in __defaults__
(position [0]
in __defaults__
) and you'll see how these are indeed refering to the same list instance:
>>> def func(a = []):
... a.append(5)
... return id(a)
>>>
>>> id(func.__defaults__[0]) == func()
True
All with the power of introspection!
* To verify that Python evaluates the default arguments during compilation of the function, try executing the following:
def bar(a=input('Did you just see me without calling the function?')):
pass # use raw_input in Py2
as you'll notice, input()
is called before the process of building the function and binding it to the name bar
is made.
id(...)
needed for that last verification, or would the is
operator answer the same question? - das-g
is
would do just fine, I just used id(val)
because I think it might be more intuitive. - Dimitris Fasarakis Hilliard
None
as the default severely limits the usefulness of the __defaults__
introspection, so I don't think that works well as a defense of having __defaults__
work the way it does. Lazy-evaluation would do more to keep function defaults useful from both sides. - Brilliand
I used to think that creating the objects at runtime would be the better approach. I'm less certain now, since you do lose some useful features, though it may be worth it regardless simply to prevent newbie confusion. The disadvantages of doing so are:
1. Performance
def foo(arg=something_expensive_to_compute())):
...
If call-time evaluation is used, then the expensive function is called every time your function is used without an argument. You'd either pay an expensive price on each call, or need to manually cache the value externally, polluting your namespace and adding verbosity.
2. Forcing bound parameters
A useful trick is to bind parameters of a lambda to the current binding of a variable when the lambda is created. For example:
funcs = [ lambda i=i: i for i in range(10)]
This returns a list of functions that return 0,1,2,3... respectively. If the behaviour is changed, they will instead bind i
to the call-time value of i, so you would get a list of functions that all returned 9
.
The only way to implement this otherwise would be to create a further closure with the i bound, ie:
def make_func(i): return lambda: i
funcs = [make_func(i) for i in range(10)]
3. Introspection
Consider the code:
def foo(a='test', b=100, c=[]):
print a,b,c
We can get information about the arguments and defaults using the inspect
module, which
>>> inspect.getargspec(foo)
(['a', 'b', 'c'], None, None, ('test', 100, []))
This information is very useful for things like document generation, metaprogramming, decorators etc.
Now, suppose the behaviour of defaults could be changed so that this is the equivalent of:
_undefined = object() # sentinel value
def foo(a=_undefined, b=_undefined, c=_undefined)
if a is _undefined: a='test'
if b is _undefined: b=100
if c is _undefined: c=[]
However, we've lost the ability to introspect, and see what the default arguments are. Because the objects haven't been constructed, we can't ever get hold of them without actually calling the function. The best we could do is to store off the source code and return that as a string.
if this shows as code, then you can use backticks to prevent it
- Brian
Simplicity: The behavior is simple in the following sense: Most people fall into this trap only once, not several times.
Consistency: Python always passes objects, not names. The default parameter is, obviously, part of the function heading (not the function body). It therefore ought to be evaluated at module load time (and only at module load time, unless nested), not at function call time.
Usefulness: As Frederik Lundh points out in his explanation of "Default Parameter Values in Python" [1], the current behavior can be quite useful for advanced programming. (Use sparingly.)
Sufficient documentation: In the most basic Python documentation, the tutorial, the issue is loudly announced as an "Important warning" in the first subsection of Section "More on Defining Functions" [2]. The warning even uses boldface, which is rarely applied outside of headings. RTFM: Read the fine manual.
Meta-learning: Falling into the trap is actually a very helpful moment (at least if you are a reflective learner), because you will subsequently better understand the point "Consistency" above and that will teach you a great deal about Python.
This behavior is easy explained by:
So:
def x(a=0, b=[], c=[], d=0):
a = a + 1
b = b + [1]
c.append(1)
print a, b, c
a
doesn't change - every assignment call creates new int object - new object is printedb
doesn't change - new array is build from default value and printedc
changes - operation is performed on same object - and it is printed__iadd__
, but it doesn't work with int. Of course. :-) - Veky
1) The so-called problem of "Mutable Default Argument" is in general a special example demonstrating that:
"All functions with this problem suffer also from similar side effect problem on the actual parameter,"
That is against the rules of functional programming, usually undesiderable and should be fixed both together.
Example:
def foo(a=[]): # the same problematic function
a.append(5)
return a
>>> somevar = [1, 2] # an example without a default parameter
>>> foo(somevar)
[1, 2, 5]
>>> somevar
[1, 2, 5] # usually expected [1, 2]
Solution: a copy
An absolutely safe solution is to copy
or deepcopy
the input object first and then to do whatever with the copy.
def foo(a=[]):
a = a[:] # a copy
a.append(5)
return a # or everything safe by one line: "return a + [5]"
Many builtin mutable types have a copy method like some_dict.copy()
or some_set.copy()
or can be copied easy like somelist[:]
or list(some_list)
. Every object can be also copied by copy.copy(any_object)
or more thorough by copy.deepcopy()
(the latter useful if the mutable object is composed from mutable objects). Some objects are fundamentally based on side effects like "file" object and can not be meaningfully reproduced by copy.
copying
[1]
Example problem for a similar SO question [2]
class Test(object): # the original problematic class
def __init__(self, var1=[]):
self._var1 = var1
somevar = [1, 2] # an example without a default parameter
t1 = Test(somevar)
t2 = Test(somevar)
t1._var1.append([1])
print somevar # [1, 2, [1]] but usually expected [1, 2]
print t2._var1 # [1, 2, [1]] but usually expected [1, 2]
It shouldn't be neither saved in any public attribute of an instance returned by this function. (Assuming that private attributes of instance should not be modified from outside of this class or subclasses by convention. i.e. _var1
is a private attribute )
Conclusion:
Input parameters objects shouldn't be modified in place (mutated) nor they should not be binded into an object returned by the function. (If we prefere programming without side effects which is strongly recommended. see
Wiki about "side effect"
[3] (The first two paragraphs are relevent in this context.)
.)
2)
Only if the side effect on the actual parameter is required but unwanted on the default parameter then the useful solution is def ...(var1=None):
if var1 is None:
var1 = []
More..
[4]
3) In some cases is the mutable behavior of default parameters useful [5].
[1] http://effbot.org/pyfaq/how-do-i-copy-an-object-in-python.htmdef f( a = None )
construct is recommended when you really mean something else. Copying is ok, because you shouldn't mutate arguments. And when you do if a is None: a = [1, 2, 3]
, you do copy the list anyway. - koddo
some_list=()
would be more unexpected. If an optional parameter with a nonzero default value is added instead of some original constant, e.g. max_iterations=10 then you can be tempted to copy the same default value to the header of another function that calls it. If you expect the original default could be changed later and you want consistency, then the default=None is better and leave the responsibility for value to the body of the function. - hynekcer
def f(x = None)
and never discusses x = copy(x)
. I think, while the former has its own uses, it's not the first thing that should be mentioned. Again, I agree with your answer and I upvoted it. - koddo
if x is None: x = { a deeply nested structure here }
? Isn't the structure deepcopied anyway from somewhere to be bound to x
every time the func called with the default x = None
? - koddo
def f(x=None)
. - holdenweb
What you're asking is why this:
def func(a=[], b = 2):
pass
isn't internally equivalent to this:
def func(a=None, b = None):
a_default = lambda: []
b_default = lambda: 2
def actual_func(a=None, b=None):
if a is None: a = a_default()
if b is None: b = b_default()
return actual_func
func = func()
except for the case of explicitly calling func(None, None), which we'll ignore.
In other words, instead of evaluating default parameters, why not store each of them, and evaluate them when the function is called?
One answer is probably right there--it would effectively turn every function with default parameters into a closure. Even if it's all hidden away in the interpreter and not a full-blown closure, the data's got to be stored somewhere. It'd be slower and use more memory.
This actually has nothing to do with default values, other than that it often comes up as an unexpected behaviour when you write functions with mutable default values.
>>> def foo(a):
a.append(5)
print a
>>> a = [5]
>>> foo(a)
[5, 5]
>>> foo(a)
[5, 5, 5]
>>> foo(a)
[5, 5, 5, 5]
>>> foo(a)
[5, 5, 5, 5, 5]
No default values in sight in this code, but you get exactly the same problem.
The problem is that foo
is modifying a mutable variable passed in from the caller, when the caller doesn't expect this. Code like this would be fine if the function was called something like append_5
; then the caller would be calling the function in order to modify the value they pass in, and the behaviour would be expected. But such a function would be very unlikely to take a default argument, and probably wouldn't return the list (since the caller already has a reference to that list; the one it just passed in).
Your original foo
, with a default argument, shouldn't be modifying a
whether it was explicitly passed in or got the default value. Your code should leave mutable arguments alone unless it is clear from the context/name/documentation that the arguments are supposed to be modified. Using mutable values passed in as arguments as local temporaries is an extremely bad idea, whether we're in Python or not and whether there are default arguments involved or not.
If you need to destructively manipulate a local temporary in the course of computing something, and you need to start your manipulation from an argument value, you need to make a copy.
append
to change a
"in-place"). That a default mutable is not re-instantiated on each call is the "unexpected" bit... at least for me. :) - Andy Hayden
cache={}
. However, I suspect this "least astonishment" comes up is when you don't expect (or want) the function you're calling to mutate the argument. - Andy Hayden
cache={}
into it for completeness. - Mark Ransom
None
and assigning the real default if the arg is None
does not resolve that problem (I consider it an anti pattern for that reason). If you fix the other bug by avoiding mutating argument values whether or not they have defaults then you'll never notice or care about this "astonishing" behavior. - Ben
None
that I've seen is in a constructor, where you want a mutable attribute (but don't want it to be shared across all classes), of course you can .copy()
before setting it so that it can't mutate the original. - Andy Hayden
list(range(n))
. - Andy Hayden
self.foo = foo.copy()
in there anyway, what harm is it if the default value for foo
is []
? It's the copy that protects you from mutable argument woes, not setting the default to None
when you really want a default of []
. Sure you could write if foo is None: self.foo = []; else: self.foo = foo.copy()
, but why when you could replace 4 lines with a single line (which is one of the 4 you need anyway), and have the real value of the default argument be clearer in the function definition? - Ben
list(foo) = foo
as the first line, this would guarantee it wasn't mutated BUT it might be a performance hit you were unwilling to take (e.g. if it was a numpy array and hitting it with list could OOM). - Andy Hayden
self.foo = foo.copy() if foo else []
:P - Andy Hayden
Default arguments get evaluated at the time the function is compiled into a function object, at the start of the program runtime. When used by the function, multiple times by that function, they are and remain the same object in memory, and when mutated (if the object is of a mutable type) they remain mutated on consecutive calls.
They are mutated and stay mutated because they are the same object each time the function is called.
Since the list is bound to the function when the function object is compiled and instantiated, this:
def foo(mutable_default_argument=[]): # make a list the default argument
"""function that uses a list"""
is almost exactly equivalent to this:
_a_list = [] # create a list in the globals
def foo(mutable_default_argument=_a_list): # make it the default argument
"""function that uses a list"""
del _a_list # remove globals name binding
Here's a demonstration - you can verify that they are the same object each time they are referenced by
example.py
print('1. Global scope being evaluated')
def create_list():
'''noisily create a list for usage as a kwarg'''
l = []
print('3. list being created and returned, id: ' + str(id(l)))
return l
print('2. example_function about to be compiled to an object')
def example_function(default_kwarg1=create_list()):
print('appending "a" in default default_kwarg1')
default_kwarg1.append("a")
print('list with id: ' + str(id(default_kwarg1)) +
' - is now: ' + repr(default_kwarg1))
print('4. example_function compiled: ' + repr(example_function))
if __name__ == '__main__':
print('5. calling example_function twice!:')
example_function()
example_function()
and running it with python example.py
:
1. Global scope being evaluated
2. example_function about to be compiled to an object
3. list being created and returned, id: 140502758808032
4. example_function compiled: <function example_function at 0x7fc9590905f0>
5. calling example_function twice!:
appending "a" in default default_kwarg1
list with id: 140502758808032 - is now: ['a']
appending "a" in default default_kwarg1
list with id: 140502758808032 - is now: ['a', 'a']
This order of execution is frequently confusing to new users of Python. If you understand the Python execution model, then it becomes quite expected.
But this is why the usual instruction to new users is to create their default arguments like this instead:
def example_function_2(default_kwarg=None):
if default_kwarg is None:
default_kwarg = []
This uses the None singleton as a sentinel object to tell the function whether or not we've gotten an argument other than the default. If we get no argument, then we actually want to use a new empty list, []
, as the default.
As the tutorial section on control flow [1] says:
[1] https://docs.python.org/tutorial/controlflow.html#default-argument-valuesIf you don’t want the default to be shared between subsequent calls, you can write the function like this instead:
def f(a, L=None): if L is None: L = [] L.append(a) return L
The shortest answer would probably be "definition is execution", therefore the whole argument makes no strict sense. As a more contrived example, you may cite this:
def a(): return []
def b(x=a()):
print x
Hopefully it's enough to show that not executing the default argument expressions at the execution time of the def
statement isn't easy or doesn't make sense, or both.
I agree it's a gotcha when you try to use default constructors, though.
Already busy topic, but from what I read here, the following helped me realizing how it's working internally:
def bar(a=[]):
print id(a)
a = a + [1]
print id(a)
return a
>>> bar()
4484370232
4484524224
[1]
>>> bar()
4484370232
4484524152
[1]
>>> bar()
4484370232 # Never change, this is 'class property' of the function
4484523720 # Always a new object
[1]
>>> id(bar.func_defaults[0])
4484370232
a = a + [1]
overloads a
... consider changing it to b = a + [1] ; print id(b)
and add a line a.append(2)
. That will make it more obvious that +
on two lists always creates a new list (assigned to b
), while a modified a
can still have the same id(a)
. - Jörn Hees
It's a performance optimization. As a result of this functionality, which of these two function calls do you think is faster?
def print_tuple(some_tuple=(1,2,3)):
print some_tuple
print_tuple() #1
print_tuple((1,2,3)) #2
I'll give you a hint. Here's the disassembly (see http://docs.python.org/library/dis.html):
#
10 LOAD_GLOBAL 0 (print_tuple)
3 CALL_FUNCTION 0
6 POP_TOP
7 LOAD_CONST 0 (None)
10 RETURN_VALUE
#
2 0 LOAD_GLOBAL 0 (print_tuple)
3 LOAD_CONST 4 ((1, 2, 3))
6 CALL_FUNCTION 1
9 POP_TOP
10 LOAD_CONST 0 (None)
13 RETURN_VALUE
I doubt the experienced behavior has a practical use (who really used static variables in C, without breeding bugs ?)
As you can see, there is a performance benefit when using immutable default arguments. This can make a difference if it's a frequently called function or the default argument takes a long time to construct. Also, bear in mind that Python isn't C. In C you have constants that are pretty much free. In Python you don't have this benefit.
This behavior is not surprising if you take the following into consideration:
The role of (2) has been covered extensively in this thread. (1) is likely the astonishment causing factor, as this behavior is not "intuitive" when coming from other languages.
(1) is described in the Python tutorial on classes [1]. In an attempt to assign a value to a read-only class attribute:
...all variables found outside of the innermost scope are read-only (an attempt to write to such a variable will simply create a new local variable in the innermost scope, leaving the identically named outer variable unchanged).
Look back to the original example and consider the above points:
def foo(a=[]):
a.append(5)
return a
Here foo
is an object and a
is an attribute of foo
(available at foo.func_defs[0]
). Since a
is a list, a
is mutable and is thus a read-write attribute of foo
. It is initialized to the empty list as specified by the signature when the function is instantiated, and is available for reading and writing as long as the function object exists.
Calling foo
without overriding a default uses that default's value from foo.func_defs
. In this case, foo.func_defs[0]
is used for a
within function object's code scope. Changes to a
change foo.func_defs[0]
, which is part of the foo
object and persists between execution of the code in foo
.
Now, compare this to the example from the documentation on emulating the default argument behavior of other languages [2], such that the function signature defaults are used every time the function is executed:
def foo(a, L=None):
if L is None:
L = []
L.append(a)
return L
Taking (1) and (2) into account, one can see why this accomplishes the desired behavior:
foo
function object is instantiated, foo.func_defs[0]
is set to None
, an immutable object.L
in the function call), foo.func_defs[0]
(None
) is available in the local scope as L
.L = []
, the assignment cannot succeed at foo.func_defs[0]
, because that attribute is read-only.L
is created in the local scope and used for the remainder of the function call. foo.func_defs[0]
thus remains unchanged for future invocations of foo
.It may be true that:
it is entirely consistent to hold to both of the features above and still make another point:
The other answers, or at least some of them either make points 1 and 2 but not 3, or make point 3 and downplay points 1 and 2. But all three are true.
It may be true that switching horses in midstream here would be asking for significant breakage, and that there could be more problems created by changing Python to intuitively handle Stefano's opening snippet. And it may be true that someone who knew Python internals well could explain a minefield of consequences. However,
The existing behavior is not Pythonic, and Python is successful because very little about the language violates the principle of least astonishment anywhere near this badly. It is a real problem, whether or not it would be wise to uproot it. It is a design flaw. If you understand the language much better by trying to trace out the behavior, I can say that C++ does all of this and more; you learn a lot by navigating, for instance, subtle pointer errors. But this is not Pythonic: people who care about Python enough to persevere in the face of this behavior are people who are drawn to the language because Python has far fewer surprises than other language. Dabblers and the curious become Pythonistas when they are astonished at how little time it takes to get something working--not because of a design fl--I mean, hidden logic puzzle--that cuts against the intuitions of programmers who are drawn to Python because it Just Works.
x=[]
means "create an empty list object, and bind the name 'x' to it." So, in def f(x=[])
, an empty list is also created. It doesn't always get bound to x, so instead it gets bound to the default surrogate. Later when f() is called, the default is hauled out and bound to x. Since it was the empty list itself that was squirreled away, that same list is the only thing available to bind to x, whether anything has been stuck inside it or not. How could it be otherwise? - Jerry B
x=[]
into byte code is compiling. And that code will have to be executed for any default value that isn't an immutable literal (anything that doesn't provably give the same result every time it's executed). - Jerry B
A simple workaround using None
>>> def bar(b, data=None):
... data = data or []
... data.append(b)
... return data
...
>>> bar(3)
[3]
>>> bar(3)
[3]
>>> bar(3)
[3]
>>> bar(3, [34])
[34, 3]
>>> bar(3, [34])
[34, 3]
I am going to demonstrate an alternative structure to pass a default list value to a function (it works equally well with dictionaries).
As others have extensively commented, the list parameter is bound to the function when it is defined as opposed to when it is executed. Because lists and dictionaries are mutable, any alteration to this parameter will affect other calls to this function. As a result, subsequent calls to the function will receive this shared list which may have been altered by any other calls to the function. Worse yet, two parameters are using this function's shared parameter at the same time oblivious to the changes made by the other.
Wrong Method (probably...):
def foo(list_arg=[5]):
return list_arg
a = foo()
a.append(6)
>>> a
[5, 6]
b = foo()
b.append(7)
# The value of 6 appended to variable 'a' is now part of the list held by 'b'.
>>> b
[5, 6, 7]
# Although 'a' is expecting to receive 6 (the last element it appended to the list),
# it actually receives the last element appended to the shared list.
# It thus receives the value 7 previously appended by 'b'.
>>> a.pop()
7
You can verify that they are one and the same object by using id
:
>>> id(a)
5347866528
>>> id(b)
5347866528
Per Brett Slatkin's "Effective Python: 59 Specific Ways to Write Better Python", Item 20: Use None
and Docstrings to specify dynamic default arguments (p. 48)
The convention for achieving the desired result in Python is to provide a default value of
None
and to document the actual behaviour in the docstring.
This implementation ensures that each call to the function either receives the default list or else the list passed to the function.
Preferred Method:
def foo(list_arg=None):
"""
:param list_arg: A list of input values.
If none provided, used a list with a default value of 5.
"""
if not list_arg:
list_arg = [5]
return list_arg
a = foo()
a.append(6)
>>> a
[5, 6]
b = foo()
b.append(7)
>>> b
[5, 7]
c = foo([10])
c.append(11)
>>> c
[10, 11]
There may be legitimate use cases for the 'Wrong Method' whereby the programmer intended the default list parameter to be shared, but this is more likely the exception than the rule.
The solutions here are:
None
as your default value (or a nonce object
), and switch on that to create your values at runtime; orlambda
as your default parameter, and call it within a try block to get the default value (this is the sort of thing that lambda abstraction is for).The second option is nice because users of the function can pass in a callable, which may be already existing (such as a type
)
You can get round this by replacing the object (and therefore the tie with the scope):
def foo(a=[]):
a = list(a)
a.append(5)
return a
Ugly, but it works.
When we do this:
def foo(a=[]):
...
... we assign the argument a
to an unnamed list, if the caller does not pass the value of a.
To make things simpler for this discussion, let's temporarily give the unnamed list a name. How about pavlo
?
def foo(a=pavlo):
...
At any time, if the caller doesn't tell us what a
is, we reuse pavlo
.
If pavlo
is mutable (modifiable), and foo
ends up modifying it, an effect we notice the next time foo
is called without specifying a
.
So this is what you see (Remember, pavlo
is initialized to []):
>>> foo()
[5]
Now, pavlo
is [5].
Calling foo()
again modifies pavlo
again:
>>> foo()
[5, 5]
Specifying a
when calling foo()
ensures pavlo
is not touched.
>>> ivan = [1, 2, 3, 4]
>>> foo(a=ivan)
[1, 2, 3, 4, 5]
>>> ivan
[1, 2, 3, 4, 5]
So, pavlo
is still [5, 5]
.
>>> foo()
[5, 5, 5]
I sometimes exploit this behavior as an alternative to the following pattern:
singleton = None
def use_singleton():
global singleton
if singleton is None:
singleton = _make_singleton()
return singleton.use_me()
If singleton
is only used by use_singleton
, I like the following pattern as a replacement:
# _make_singleton() is called only once when the def is executed
def use_singleton(singleton=_make_singleton()):
return singleton.use_me()
I've used this for instantiating client classes that access external resources, and also for creating dicts or lists for memoization.
Since I don't think this pattern is well known, I do put a short comment in to guard against future misunderstandings.
_make_singleton
at def time in the default argument example, but at call time in the global example. A true substitution would use some sort of mutable box for the default argument value, but the addition of the argument makes an opportunity to pass alternate values. - Yann Vernier
Every other answer explains why this is actually a nice and desired behavior, or why you shouldn't be needing this anyway. Mine is for those stubborn ones who want to exercise their right to bend the language to their will, not the other way around.
We will "fix" this behavior with a decorator that will copy the default value instead of reusing the same instance for each positional argument left at its default value.
import inspect
from copy import deepcopy # copy would fail on deep arguments like nested dicts
def sanify(function):
def wrapper(*a, **kw):
# store the default values
defaults = inspect.getargspec(function).defaults # for python2
# construct a new argument list
new_args = []
for i, arg in enumerate(defaults):
# allow passing positional arguments
if i in range(len(a)):
new_args.append(a[i])
else:
# copy the value
new_args.append(deepcopy(arg))
return function(*new_args, **kw)
return wrapper
Now let's redefine our function using this decorator:
@sanify
def foo(a=[]):
a.append(5)
return a
foo() # '[5]'
foo() # '[5]' -- as desired
This is particularly neat for functions that take multiple arguments. Compare:
# the 'correct' approach
def bar(a=None, b=None, c=None):
if a is None:
a = []
if b is None:
b = []
if c is None:
c = []
# finally do the actual work
with
# the nasty decorator hack
@sanify
def bar(a=[], b=[], c=[]):
# wow, works right out of the box!
It's important to note that the above solution breaks if you try to use keyword args, like so:
foo(a=[4])
The decorator could be adjusted to allow for that, but we leave this as an exercise for the reader ;)
{"grandparent": {"parent": {"child": "value"}}}
. Only the top-level dictionary gets copied by value, the other dictionaries get copied by reference. This problem occurs because you used copy
instead of deepcopy
- Flimm
I've read all the other answers and I'm not convinced. This design does violate the principle of least astonishment.
The defaults could have been designed to be evaluated when the function is called, rather than when the function is defined. This is how Javascript does it:
function foo(a=[]) {
a.push(5);
return a;
}
console.log(foo()); // [5]
console.log(foo()); // [5]
console.log(foo()); // [5]
As further evidence that this is a design flaw, Python core developers are currently discussing introducing new syntax to fix this problem. See this article: Late-bound argument defaults for Python [1].
For even more evidence that this a design flaw, if you Google "Python gotchas", this design is mentioned as a gotcha, usually the first gotcha in the list, in the first 9 Google results ( 1 [2], 2 [3], 3 [4], 4 [5], 5 [6], 6 [7], 7 [8], 8 [9], 9 [10]). In contrast, if you Google "Javascript gotchas", the behaviour of default arguments in Javascript is not mentioned as a gotcha even once.
Gotchas, by definition, violate the principle of least astonishment. They astonish. Given there are superiour designs for the behaviour of default argument values, the inescapable conclusion is that Python's behaviour here represents a design flaw.
I say this as someone who loves Python. We can be fans of Python, and still admit that everyone who is unpleasantly surprised by this aspect of Python is unpleasantly surprised because it is a genuine "gotcha".
[1] https://lwn.net/Articles/875441/def
is a statement that is executed at module load time (for a top-level function). The default argument definition is part of that statement and therefore ought to be executed at load time as well -- and any other behavior would also be surprising. - Lutz Prechelt
def
statement is executed at module load time, that does not mean that the default arg values should be executed at module load time too. Take the statement foobar = lambda x: print("hi")
. Would you be surprised if hi
got printed at module load time? I know I would be, even though this statement is executed at module load time. Intuitively, we expect that the print
part of this statement would only be executed once the lambda is called. Likewise, as evidenced by people's well-documented surprised, we expect the default value to be executed when called - Flimm
This "bug" gave me a lot of overtime work hours! But I'm beginning to see a potential use of it (but I would have liked it to be at the execution time, still)
I'm gonna give you what I see as a useful example.
def example(errors=[]):
# statements
# Something went wrong
mistake = True
if mistake:
tryToFixIt(errors)
# Didn't work.. let's try again
tryToFixItAnotherway(errors)
# This time it worked
return errors
def tryToFixIt(err):
err.append('Attempt to fix it')
def tryToFixItAnotherway(err):
err.append('Attempt to fix it by another way')
def main():
for item in range(2):
errors = example()
print '\n'.join(errors)
main()
prints the following
Attempt to fix it
Attempt to fix it by another way
Attempt to fix it
Attempt to fix it by another way
errors
as a parameter rather than starting from scratch every time? - Mark Ransom
Just change the function to be:
def notastonishinganymore(a = []):
'''The name is just a joke :)'''
a = a[:]
a.append(5)
return a
This is not a design flaw. Anyone who trips over this is doing something wrong.
There are 3 cases I see where you might run into this problem:
cache={}
, and you wouldn't be expected to call the function with an actual argument at all.The example in the question could fall into category 1 or 3. It's odd that it both modifies the passed list and returns it; you should pick one or the other.
cache={}
pattern is really an interview-only solution, in real code you probably want @lru_cache
! - Andy Hayden
x = modified(x)
is a well known idiom in Python, and would work well with a default argument - as long as you make sure to copy the input before you start making modifications. - Mark Ransom
None
and use the if x is None:
idiom. Realize that Python and Javascript are not the same. - Mark Ransom
TLDR: Define-time defaults are consistent and strictly more expressive.
Defining a function affects two scopes: the defining scope containing the function, and the execution scope contained by the function. While it is pretty clear how blocks map to scopes, the question is where def <name>(<args=defaults>):
belongs to:
... # defining scope
def name(parameter=default): # ???
... # execution scope
The def name
part must evaluate in the defining scope - we want name
to be available there, after all. Evaluating the function only inside itself would make it inaccessible.
Since parameter
is a constant name, we can "evaluate" it at the same time as def name
. This also has the advantage it produces the function with a known signature as name(parameter=...):
, instead of a bare name(...):
.
Now, when to evaluate default
?
Consistency already says "at definition": everything else of def <name>(<args=defaults>):
is best evaluated at definition as well. Delaying parts of it would be the astonishing choice.
The two choices are not equivalent, either: If default
is evaluated at definition time, it can still affect execution time. If default
is evaluated at execution time, it cannot affect definition time. Choosing "at definition" allows expressing both cases, while choosing "at execution" can express only one:
def name(parameter=defined): # set default at definition time
...
def name(parameter=default): # delay default until execution time
parameter = default if parameter is None else parameter
...
def <name>(<args=defaults>):
is best evaluated at definition as well." I don't think the conclusion follows from the premise. Just because two things are on the same line doesn't mean they should be evaluated in the same scope. default
is a different thing than the rest of the line: it's an expression. Evaluating an expression is a very different process from defining a function. - LarsH
def
) or expression (lambda
) does not change that creating a function means evaluation -- especially of its signature. And defaults are part of a function's signature. That does not mean defaults have to be evaluated immediately -- type hints may not, for example. But it certainly suggests they should unless there is a good reason not to. - MisterMiyagi
I think the answer to this question lies in how python pass data to parameter (pass by value or by reference), not mutability or how python handle the "def" statement.
A brief introduction. First, there are two type of data types in python, one is simple elementary data type, like numbers, and another data type is objects. Second, when passing data to parameters, python pass elementary data type by value, i.e., make a local copy of the value to a local variable, but pass object by reference, i.e., pointers to the object.
Admitting the above two points, let's explain what happened to the python code. It's only because of passing by reference for objects, but has nothing to do with mutable/immutable, or arguably the fact that "def" statement is executed only once when it is defined.
[] is an object, so python pass the reference of [] to a
, i.e., a
is only a pointer to [] which lies in memory as an object. There is only one copy of [] with, however, many references to it. For the first foo(), the list [] is changed to
1
[1] by append method. But Note that there is only one copy of the list object and this object now becomes
1
[2]. When running the second foo(), what effbot webpage says (items is not evaluated any more) is wrong. a
is evaluated to be the list object, although now the content of the object is
1
[3]. This is the effect of passing by reference! The result of foo(3) can be easily derived in the same way.
To further validate my answer, let's take a look at two additional codes.
====== No. 2 ========
def foo(x, items=None):
if items is None:
items = []
items.append(x)
return items
foo(1) #return [1]
foo(2) #return [2]
foo(3) #return [3]
[]
is an object, so is None
(the former is mutable while the latter is immutable. But the mutability has nothing to do with the question). None is somewhere in the space but we know it's there and there is only one copy of None there. So every time foo is invoked, items is evaluated (as opposed to some answer that it is only evaluated once) to be None, to be clear, the reference (or the address) of None. Then in the foo, item is changed to [], i.e., points to another object which has a different address.
====== No. 3 =======
def foo(x, items=[]):
items.append(x)
return items
foo(1) # returns [1]
foo(2,[]) # returns [2]
foo(3) # returns [1,3]
The invocation of foo(1) make items point to a list object [] with an address, say, 11111111. the content of the list is changed to
1
[4] in the foo function in the sequel, but the address is not changed, still 11111111. Then foo(2,[]) is coming. Although the [] in foo(2,[]) has the same content as the default parameter [] when calling foo(1), their address are different! Since we provide the parameter explicitly, items
has to take the address of this new []
, say 2222222, and return it after making some change. Now foo(3) is executed. since only x
is provided, items has to take its default value again. What's the default value? It is set when defining the foo function: the list object located in 11111111. So the items is evaluated to be the address 11111111 having an element 1. The list located at 2222222 also contains one element 2, but it is not pointed by items any more. Consequently, An append of 3 will make items
[1,3].
From the above explanations, we can see that the effbot [5] webpage recommended in the accepted answer failed to give a relevant answer to this question. What is more, I think a point in the effbot webpage is wrong. I think the code regarding the UI.Button is correct:
for i in range(10):
def callback():
print "clicked button", i
UI.Button("button %s" % i, callback)
Each button can hold a distinct callback function which will display different value of i
. I can provide an example to show this:
x=[]
for i in range(10):
def callback():
print(i)
x.append(callback)
If we execute x[7]()
we'll get 7 as expected, and x[9]()
will gives 9, another value of i
.
x[7]()
is 9
. - Duncan
Documentation failure
def fn(<mutable object> = [manifest constant]):
This python syntax is for an optional initializer for a mutable object. Calling the manifest constant a default value is both wrong and confusing. It is particularly unfortunate that the official python documentation uses the misleading description, and that the painfully misleading term has been copied across the web, including in the question here.
The problem would not be completely solved by correcting the documentation: people would still want python to implement a default value for mutable objects. But it would be a start.
Python strongly respects the principle of
command-query separation
[1]. For example, ordinary assignment is considered a statement, not an expression, and thus cannot be used as a sub-expression; and
many built-in and standard library methods return
None
[2], rather than the object, when they work primarily by modifying the object.
We can make a simple argument against using mutable default values for parameters, as follows:
There are
two ways to get information out of the function
[3]: by return
ing it, or by modifying some variable in an enclosing scope.
However, the only sane, reusable way to "modify a variable in an enclosing scope" is to modify one of the parameters. Modifying the function itself makes the interface much more awkward (and could cause problems for recursive code). Modifying anything else requires the function and the caller to share that context; this means polluting a global namespace and, again, creating an awkward and highly non-standard interface.
The point of default parameter values is to be able to call the function without the corresponding argument; therefore, we must design for the case where that happens.
Suppose we call the function without the corresponding argument, so that that the default is used. How will we signal the change? The object for the default value is not in scope for the caller, and it is not an attribute of the function; therefore, it is not readily accessible to the caller. Therefore, we can't readily get the information back via the now-mutated default.
Therefore, in order to communicate a result, we would have to return
some information instead.
In order for the interface to be consistent, and avoid complicated special cases for using it, the function should therefore consistently return a calculated result.
But because of command-query separation, this therefore means that the function should not modify the parameter. We cannot do both, because then "asking the question could change the answer".
Since the function will not modify the parameter, it will not modify the default-value object.
But if the object will not be modified, there is no reason for it to be mutable. Using a mutable type misleadingly implies an intent to mutate the value.
In practice, people sometimes write default arguments like {}
even when the code will not mutate them. It's easy enough to replace []
with ()
, but it's not trivial to express "an empty, immutable mapping". Of course, using None
(which is immutable) as a sentinel and then explicitly checking for that value, neatly sidesteps the problem.
There is a simple way to understand why this happens.
Python executes code, from top to bottom, in a namespace.
The 'internals' just embody of this rule.
The reason for this choice is to "let the language fit in your head". All the odd corner cases tend to simplify to executing code in a namespace: default immutables, nested functions, classes (with a little patch-up when done compiling), the self argument, etc. Similarly, complex syntax could be written in simple syntax: a.foo(...)
is just a.lookup('foo').__call__(a,...)
. This works with list comprehensions; decorators; metaclasses; and more. This gives you a near perfect view of the strange corners. The language fits in your head.
You should keep at it. Learning Python has a period of railing at the language, but it gets comfortable. It's the only language I've worked in that gets simpler the more you look at corner cases.
Keep Hacking! Keep notes.
For your specific code, in too much detail:
def foo(a=[]):
a.append(5)
return a
foo()
is one statement, equivalent to:
(a=[])
right now, as we go. The []
is the default value of argument a. It is of type list, as []
always is.:
into Python bytecode and stick it in another list.Then, it goes to the next line, foo()
.
append
does not create a new list, so the old one is modified.
[5]
." I'm a Python novice, and I wouldn't expect this, because obviouslyfoo([1])
will return[1, 5]
, not[5]
. What you meant to say is that a novice would expect the function called with no parameter will always return[5]
. - symplectomorphicif L is None:
needed? I removed this test and it made no difference - sdaffa23fdsfL
is notNone
, You wouldn't want to override the incoming value of that parameter. - nehem[]
is syntactic sugar for a function call which returns a value. To the beginner, it looks like "the default argument is a new empty list", not "the default argument is a reference to a particular list, initially empty". The "language bug" that I'm seeing here is that python is designed so that people can get by without thinking about references for a lot longer than in other languages, and when they turn up, it's startling and disconcerting. However, this is usually considered a feature. - Jon Kiparskylist()
in place of[]
. - wjandrealist()
as they do for[]
- I agree that settinglist()
as a parameter for a function would have exactly the same problem as using[]
, but I'm not sure that users would expect the former to work, where they clearly expect the latter to work. - Jon Kiparsky__get__
, or passing it as a default arg. The latter is much cleaner, more concise, and the standard idiom. - BadZenNone
fix the problem?" question is not really about mutable default arguments anyway: it is primarily yet another question about assignment vs. mutation.) - Karl Knechtel