API: chained reshaping ops

[jreback]

accept callables

• .query (not changed, see ENH: Allow where/mask/Indexers to accept callable #12539)
• .where/.mask (ENH: Allow where/mask/Indexers to accept callable #12539)
• .loc (and indexers) (ENH: Allow where/mask/Indexers to accept callable #12539)

In [4]: df = DataFrame({'A' : [1,2,3,4], 'B' : ['a',np.nan,'b','a']})

In [5]: df
Out[5]: 
   A    B
0  1    a
1  2  NaN
2  3    b
3  4    a

an operation that changes the shape of the DataFrame

In [9]: res = df.dropna()

In [10]: res[res.B=='a']
Out[10]: 
   A  B
0  1  a
3  4  a

can be done like this

In [8]: df.dropna().pipe(lambda x: x[x.B=='a'])
Out[8]: 
   A  B
0  1  a
3  4  a

SQL calls this select, which pandas has, but both select/filter are used for filtering LABELS (and not data).

I suppose making this work:

df.dropna().loc[lambda x: x[x.B=='a']] is maybe a slight enhancement of this

any thoughts?

[jreback]

cc @jorisvandenbossche @TomAugspurger @sinhrks @shoyer

FYI @TomAugspurger I really do like .pipe & chaining!

comes from this example

(tidy
     .dropna()
     .pipe(lambda df: df[df.team == 'Los Angeles Lakers'])
     .pipe(sns.FacetGrid, col='team', hue='team')
     .map(sns.barplot, "variable", "rest")
 )

[jreback]

forgot about .query which is a nice soln for this actually

In [63]: df.dropna().query('B=="a"')            
Out[63]: 
   A  B
0  1  a
3  4  a

[shoyer]

Yes, .query works but I hate coding in strings. I would be supportive of accepting a lambda in query, e.g., df.query(lambda df: df.B == 'a')

[jreback]

yeh, then of course

df[lambda x: x.B == 'a']
df.loc[lambda x: x.B == 'a']

for consistency

ok, then it allows easy chaining based on values, which is nice to do (and then is consistent with how .assign works, e.g. accepts an expression or a lambda)

[shoyer]

Yep, putting support for functions in indexing makes sense to me.

[max-sixty]

Would be nice if this also worked on Series (unlike query which currently just works on DFs).

And +1 for @shoyer's suggestion re extending query for this: df.query(lambda df: df.B == 'a'), despite the inability to set those values.

[kawochen]

@MaximilianR I'm using your example from #12226:
What's wrong with the following?

In [15]: pd.Series(range(10)).mul(5).pipe(lambda x: x**2).pipe(lambda x: x-500).pipe(lambda x: x[x>200])
Out[15]:
6     400
7     725
8    1100
9    1525
dtype: int64

[max-sixty]

@kawochen Ah - that's very nice. Not sure how I missed that. Thanks!