share
Stack OverflowGroupby with User Defined Functions Pandas
[+38] [1] Woody Pride
[2013-10-27 07:41:51]
[ python pandas ]
[ https://stackoverflow.com/questions/19615760/groupby-with-user-defined-functions-pandas ]

I understand that passing a function as a group key calls the function once per index value with the return values being used as the group names. What I can't figure out is how to call the function on column values.

So I can do this:

people = pd.DataFrame(np.random.randn(5, 5), 
                      columns=['a', 'b', 'c', 'd', 'e'],
                      index=['Joe', 'Steve', 'Wes', 'Jim', 'Travis'])
def GroupFunc(x):
    if len(x) > 3:
        return 'Group1'
    else:
        return 'Group2'

people.groupby(GroupFunc).sum()

This splits the data into two groups, one of which has index values of length 3 or less, and the other with length three or more. But how can I pass one of the column values? So for example if column d value for each index point is greater than 1. I realise I could just do the following:

people.groupby(people.a > 1).sum()

But I want to know how to do this in a user defined function for future reference.

Something like:

def GroupColFunc(x):
if x > 1:
    return 'Group1'
else:
    return 'Group2'

But how do I call this? I tried

people.groupby(GroupColFunc(people.a))

and similar variants but this does not work.

How do I pass the column values to the function? How would I pass multiple column values e.g. to group on whether people.a > people.b for example?

[+53] [2013-10-27 08:28:57] Roman Pekar [ACCEPTED]

To group by a > 1, you can define your function like:

>>> def GroupColFunc(df, ind, col):
...     if df[col].loc[ind] > 1:
...         return 'Group1'
...     else:
...         return 'Group2'
... 

An then call it like

>>> people.groupby(lambda x: GroupColFunc(people, x, 'a')).sum()
               a         b         c         d        e
Group2 -2.384614 -0.762208  3.359299 -1.574938 -2.65963

Or you can do it only with anonymous function:

>>> people.groupby(lambda x: 'Group1' if people['b'].loc[x] > people['a'].loc[x] else 'Group2').sum()
               a         b         c         d         e
Group1 -3.280319 -0.007196  1.525356  0.324154 -1.002439
Group2  0.895705 -0.755012  1.833943 -1.899092 -1.657191

As said in documentation [1], you can also group by passing Series providing a label -> group name mapping:

>>> mapping = np.where(people['b'] > people['a'], 'Group1', 'Group2')
>>> mapping
Joe       Group2
Steve     Group1
Wes       Group2
Jim       Group1
Travis    Group1
dtype: string48
>>> people.groupby(mapping).sum()
               a         b         c         d         e
Group1 -3.280319 -0.007196  1.525356  0.324154 -1.002439
Group2  0.895705 -0.755012  1.833943 -1.899092 -1.657191
[1] http://pandas.pydata.org/pandas-docs/dev/groupby.html

(1) Thanks for the thorough answer. I am working with the McKinney book on pandas, and it comes so close to being really thorough, but explanations like this I find it hard to track down. Thank you - Woody Pride
1