Set pd.options.display.max_columns=0 by default #17023

cbrnr · 2017-07-19T11:04:00Z

Update: Remove everything related to max_rows and only deal with max_columns in this PR.

Changed max_columns to 0 (automatically adapt the number of displayed columns to the actual terminal width) when run in a terminal ~~and max_rows to 20 (because I'd like to see the "whole" data frame at a glance like in R's tibble)~~.

closes Can't pd.options.display.max_columns = 0 by default? #16579
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

TomAugspurger · 2017-07-19T11:43:11Z

Could you provide some before / after screenshots? This will need some feedback from the wider community, since the visual display of DataFrames is an API grey-zone; prepare for bike-shedding 😄

We still have some situations where we can't detect the terminal width reliably. We need to make sure the output is handled as well as possible there.

I'm +1 for reducing the number of rows displayed. Typically I use 10 rows.

cbrnr · 2017-07-19T11:48:52Z

Here's the current output when printing a data frame with shape (5, 10) in a terminal with 100 characters width:

And here is the same data frame after the proposed change:

chris-b1 · 2017-07-19T22:51:58Z

I'm OK with this. I do think it needs a big note in the whatsnew, with instructions on how to change back (maybe a ref to IPython config too). Also looks like some tests that will need adjusted.

jreback · 2017-07-20T10:31:49Z

pandas/core/config_init.py

                       validator=is_instance_factory([type(None), int]))
    cf.register_option('max_categories', 8, pc_max_categories_doc,
                       validator=is_int)
    cf.register_option('max_colwidth', 50, max_colwidth_doc, validator=is_int)
-    cf.register_option('max_columns', 20, pc_max_cols_doc,
+    cf.register_option('max_columns', 0, pc_max_cols_doc,


hmm, this should be None to auto-detect (0 might do the same though)

So should I change 0 to None or leave it as is?

A quick test shows that None means there is no limit (i.e. display all columns). So I guess this should remain 0.

hmm, so 0 is NOT the same as None here ? that is very odd. can you show an example

Well, I tried setting this value to None after importing pandas, i.e.

import pandas as pd pd.options.display.max_columns = None

And this results in all columns printed out (so no ellipsis to mark skipped columns in the output). But I will try setting the value in config_init.py as well.

I can confirm that None and 0 have different meanings. None prints all columns, whereas 0 prints columns that fit within the terminal width.

jreback · 2017-07-20T10:34:07Z

yeah changing the column default to auto-detect is fine. I personally use an event smaller default for max_rows, but 20 looks fine. Pls update docs, a before/after screen shot that we can include in the what's new would be good (IOW your above ones). You will have to fix some tests.

cbrnr · 2017-07-20T12:18:23Z

Do you really want screenshots or could we mimic the old and new behavior with markdown? If you want screenshots let me modify them so that my username doesn't show up. Regarding the tests, I'll see what I can do (I certainly didn't expect to break so many tests just by changing one value 😄).

jreback · 2017-07-20T12:22:30Z

@cbrnr

the issue is that we are now auto-detecting and so the actual terminal width matters. yes certainly we can 'set' it so it works and show in mark down. I think screenshots might be more clear here though.

cbrnr · 2017-07-20T12:25:00Z

I see, I'll provide new screenshots then. Thanks for pointing out the issue, of course this makes a huge difference!

cbrnr · 2017-07-20T13:23:40Z

Also, the IPython QtConsole doesn't play nicely with pd.options.display.column_width=0:

chris-b1 · 2017-07-20T14:26:31Z

@takluyver - assuming this hasn't changed, but do you know offhand if it's still not possible to detect terminal size running in the qtconsole? Found an older SO answer from you, thanks!
https://stackoverflow.com/questions/27813132/determining-terminal-width-in-ipython-qtconsole

takluyver · 2017-07-20T14:37:41Z

No, sorry. It's a conceptual mismatch, not just a technical one. The kernel is producing output for (potentially) several frontends which may be receiving it at the moment, and for applications which may later display saved copies of that output. So questions about the shape of 'the' output area don't really make sense in the Jupyter protocol.

As I see it, the real issue is that the Qt console doesn't understand any structured way of representing a table. We turned off its HTML support because it's just too limited and tends to break richer HTML written for the notebook frontend. I have occasionally advocated for a 'simple HTML' repr option which the Qt console would display, but it's never been high priority.

In the long run, I think our plan is to make an HTML console and use QtWebkit to embed it in Qt applications. Then it should be able to display HTML tables.

cbrnr · 2017-07-21T07:57:07Z

Thanks @takluyver - so this isn't going to work until HTML tables are rendered (which would be awesome BTW). Is it possible to determine if Pandas is running in a real terminal or not? Could someone point me to the relevant code parts?

takluyver · 2017-07-25T09:52:01Z

It's possible to distinguish terminal IPython from IPython as a kernel for a Jupyter frontend, something like this:

try:
    ip = get_ipython()
except NameError:
    ... # Not IPython
else:
    if hasattr(ip, 'kernel'):
        ... # IPython as a Jupyter kernel
    else:
        ... # IPython terminal interface

jreback · 2017-09-23T16:54:23Z

@cbrnr can you rebase this and compose a note for the what's new?

cbrnr · 2017-09-24T13:40:27Z

Sure, but many tests need to be fixed and I don't know if I have the time to do that. I guess this change should be added to the API changes section?

jreback · 2017-09-24T14:25:21Z

@cbrnr this would need its own sub-section in API

yes would need to fix any tests.

cbrnr · 2017-10-18T08:05:08Z

cf #16800 #4907

cbrnr · 2017-11-07T13:51:42Z

Many tests rely on calling str on a data frame with the current default max number of columns. I'm not sure this will be easy to fix. This would be easy to fix if Pandas supported a pandasrc config file as proposed in #4907.

jreback · 2017-11-07T17:59:50Z

Many tests rely on calling str on a data frame with the current default max number of columns. I'm not sure this will be easy to fix. This would be easy to fix if Pandas supported a pandasrc config file as proposed in #4907.

nothing to do with that issue
all options already have defaults
for testing you need to setup the specific conditions for tests
generally using pd.option_context

cbrnr · 2017-11-09T12:50:54Z

OK, I've fixed almost all tests. Only 2 tests still fail, but I'm not sure if these failures are related to my changes:

pandas/tests/tseries/test_timezones.py:1290: AssertionError
pandas/tests/scalar/test_timestamp.py:1110: AssertionError

Here's the complete test output:

___________________________________ TestTimestamp.test_timestamp ___________________________________
[gw1] darwin -- Python 3.6.3 /Users/clemens/anaconda/envs/pandas_dev/bin/python
self = <pandas.tests.scalar.test_timestamp.TestTimestamp object at 0x10ac29940>

    def test_timestamp(self):
        # GH#17329
        # tz-naive --> treat it as if it were UTC for purposes of timestamp()
        ts = Timestamp.now()
        uts = ts.replace(tzinfo=utc)
        assert ts.timestamp() == uts.timestamp()
    
        tsc = Timestamp('2014-10-11 11:00:01.12345678', tz='US/Central')
        utsc = tsc.tz_convert('UTC')
        # utsc is a different representation of the same time
        assert tsc.timestamp() == utsc.timestamp()
    
        if PY3:
            # should agree with datetime.timestamp method
            dt = ts.to_pydatetime()
>           assert dt.timestamp() == ts.timestamp()
E           AssertionError: assert 1510231568.085538 == 1510235168.085538
E            +  where 1510231568.085538 = <built-in method timestamp of datetime.datetime object at 0x10c197d50>()
E            +    where <built-in method timestamp of datetime.datetime object at 0x10c197d50> = datetime.datetime(2017, 11, 9, 13, 46, 8, 85538).timestamp
E            +  and   1510235168.085538 = <built-in method timestamp of Timestamp object at 0x10c16fb10>()
E            +    where <built-in method timestamp of Timestamp object at 0x10c16fb10> = Timestamp('2017-11-09 13:46:08.085538').timestamp

pandas/tests/scalar/test_timestamp.py:1110: AssertionError
________________________________ TestTimeZones.test_replace_tzinfo _________________________________
[gw1] darwin -- Python 3.6.3 /Users/clemens/anaconda/envs/pandas_dev/bin/python
self = <pandas.tests.tseries.test_timezones.TestTimeZones object at 0x10e8f2f98>

    def test_replace_tzinfo(self):
        # GH 15683
        dt = datetime(2016, 3, 27, 1)
        tzinfo = pytz.timezone('CET').localize(dt, is_dst=False).tzinfo
    
        result_dt = dt.replace(tzinfo=tzinfo)
        result_pd = Timestamp(dt).replace(tzinfo=tzinfo)
    
        if hasattr(result_dt, 'timestamp'):  # New method in Py 3.3
            assert result_dt.timestamp() == result_pd.timestamp()
        assert result_dt == result_pd
        assert result_dt == result_pd.to_pydatetime()
    
        result_dt = dt.replace(tzinfo=tzinfo).replace(tzinfo=None)
        result_pd = Timestamp(dt).replace(tzinfo=tzinfo).replace(tzinfo=None)
    
        if hasattr(result_dt, 'timestamp'):  # New method in Py 3.3
>           assert result_dt.timestamp() == result_pd.timestamp()
E           AssertionError: assert 1459036800.0 == 1459040400.0
E            +  where 1459036800.0 = <built-in method timestamp of datetime.datetime object at 0x10e8effd0>()
E            +    where <built-in method timestamp of datetime.datetime object at 0x10e8effd0> = datetime.datetime(2016, 3, 27, 1, 0).timestamp
E            +  and   1459040400.0 = <built-in method timestamp of Timestamp object at 0x10e8fcf48>()
E            +    where <built-in method timestamp of Timestamp object at 0x10e8fcf48> = Timestamp('2016-03-27 01:00:00').timestamp

pandas/tests/tseries/test_timezones.py:1290: AssertionError

Any ideas?

jreback · 2017-11-09T12:56:12Z

@cbrnr ignore those, see #18037

python .timestamp() uses the local timezone to convert things, needs to be put into a consistent tz so it works for everyone.

cbrnr · 2017-11-09T12:59:43Z

OK cool, so let's wait if CIs come back happy (except for these 2 timezone-related ones). Could you help me with the whats_new entry (because we've agreed that this should be prominently visible)?

Also, I hope that my changes to the tests are OK, I mostly set the values for max_columns and max_rows to their old defaults 20 and 60, respectively.

jreback · 2017-11-09T13:08:40Z

whatsnew, make a new subsection. then put a screen shot of the before and one of the after. then it should read as if you are a user wanting to know whether this change will affect you (e.g. if you use ipython, the interpreter, etc).

jreback · 2017-11-09T13:08:46Z

for 0.22

cbrnr · 2017-11-09T13:37:06Z

A new subsection under "New features"? It's not really a new feature, but it doesn't fit into the other categories either.

jreback · 2017-11-09T13:40:01Z

under api breaking changes

codecov · 2017-11-09T15:29:18Z

Codecov Report

Merging #17023 into master will decrease coverage by 0.04%.
The diff coverage is 66.66%.

@@            Coverage Diff             @@
##           master   #17023      +/-   ##
==========================================
- Coverage   91.42%   91.38%   -0.05%     
==========================================
  Files         163      163              
  Lines       50068    50071       +3     
==========================================
- Hits        45776    45755      -21     
- Misses       4292     4316      +24

Flag	Coverage Δ
#multiple	`89.18% <66.66%> (-0.03%)`	⬇️
#single	`40.39% <66.66%> (-0.04%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/config_init.py	`96.09% <66.66%> (-2.26%)`	⬇️
pandas/io/gbq.py	`25% <0%> (-58.34%)`	⬇️
pandas/plotting/_converter.py	`63.38% <0%> (-1.82%)`	⬇️
pandas/core/frame.py	`97.8% <0%> (-0.1%)`	⬇️
pandas/core/groupby.py	`92.02% <0%> (-0.02%)`	⬇️
pandas/io/formats/format.py	`96.01% <0%> (ø)`	⬆️
pandas/core/generic.py	`95.72% <0%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8dac633...85d6225. Read the comment docs.

codecov · 2017-11-09T15:29:32Z

Codecov Report

Merging #17023 into master will increase coverage by 0.01%.
The diff coverage is 73.33%.

@@            Coverage Diff             @@
##           master   #17023      +/-   ##
==========================================
+ Coverage   91.82%   91.84%   +0.01%     
==========================================
  Files         152      152              
  Lines       49235    49245      +10     
==========================================
+ Hits        45212    45230      +18     
+ Misses       4023     4015       -8

Flag	Coverage Δ
#multiple	`90.23% <73.33%> (+0.01%)`	⬆️
#single	`41.89% <66.66%> (ø)`	⬆️

Impacted Files	Coverage Δ
pandas/io/formats/format.py	`98.24% <100%> (ø)`	⬆️
pandas/io/formats/terminal.py	`20.98% <66.66%> (+4.54%)`	⬆️
pandas/core/config_init.py	`99.24% <80%> (-0.76%)`	⬇️
pandas/core/arrays/categorical.py	`96.19% <0%> (-0.02%)`	⬇️
pandas/core/indexes/datetimes.py	`95.73% <0%> (-0.01%)`	⬇️
pandas/core/indexes/period.py	`92.61% <0%> (ø)`	⬆️
pandas/core/strings.py	`98.32% <0%> (ø)`	⬆️
pandas/core/frame.py	`97.18% <0%> (ø)`	⬆️
pandas/core/dtypes/missing.py	`91.07% <0%> (ø)`	⬆️
pandas/core/generic.py	`95.85% <0%> (ø)`	⬆️
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6c0c277...f795914. Read the comment docs.

jorisvandenbossche · 2018-03-27T11:34:43Z

I moved the images to _static. Do I need to change anything when I refer to them, or does

.. image:: print_df_old.png

keep working?

I think you need to add _static/ to the path (at least, that's how we do it for all other images -> yes, the path it relative to the source file, or absolute to the main source directory)

cbrnr · 2018-03-27T11:36:49Z

Regarding the deleted images (that I locally moved to doc/source/_static), this path is in .gitignore, which is why they are gone. How should I proceed?

jorisvandenbossche · 2018-03-27T11:37:47Z

@cbrnr you need to add them by force (git add --force) to overwrite this ignore file (we ignore it because sphinx adds more images there that should be ignored)

jorisvandenbossche · 2018-03-27T11:38:35Z

we ignore it because sphinx adds more images there that should be ignored

We could also decide to move our actual images somewhere else, to not have this confusion, but that's for another PR.

cbrnr · 2018-03-27T11:41:21Z

Got it, the images are back.

cbrnr · 2018-03-27T12:32:31Z

What do you mean with adding _static/ to the path? Do I need to modify the link I use in the .rst file?

jorisvandenbossche · 2018-03-27T12:40:19Z

What do you mean with adding _static/ to the path? Do I need to modify the link I use in the .rst file?

Yes. You can always check if the images are included in the output with python doc/make.py --single whatsnew

cbrnr · 2018-03-27T13:00:48Z

Nice, thanks! Images are now correctly embedded.

cbrnr · 2018-03-27T13:02:27Z

I think I already asked that, but is it possible to see the HTML docs built by a CI service? I know that other projects use CircleCI for this purpose (so that it is not necessary to set up everything locally).

jorisvandenbossche · 2018-03-27T13:18:32Z

I think I already asked that, but is it possible to see the HTML docs built by a CI service? I know that other projects use CircleCI for this purpose (so that it is not necessary to set up everything locally).

No, it is currently not possible. Open issue about this: #17921

jorisvandenbossche · 2018-03-27T13:51:06Z

doc/source/whatsnew/v0.23.0.txt

+
+  pd.options.display.max_columns = 20
+
+.. _whatsnew_0230.api:


I think this one sneaked in due to merge conflict? (anyhow it can be removed)

I'm sorry, what do you mean?

According to the diff, you added this line. But this should not be added (therefore I assumed you added it by accident while updating against master with rebasing/merging). But so you can just remove this line.

You mean line 690 (.. _whatsnew_0230.api:)?

yes, that's the line on which I commented. There is no header following for which that link would make sense (there is actually already another link label on line 692)

I thought there should be a section header before introducing the subsections. At least that's how it is done with .. _whatsnew_0230.api_breaking: in line 350 (but there is a heading after that). In any case, I'm happy to delete it.

Yes, that is correct. But I don't understand the relation with this line? This link is just floating with no section or subsection header following it. You already have a header with link at line 661 - 664 ?

It's for the section below:

.. _whatsnew_0230.api: .. _whatsnew_0230.api.datetimelike: Datetimelike API Changes ^^^^^^^^^^^^^^^^^^^^^^^^

But that header already has the "whatsnew_0230.api.datetimelike" label, it does not need two labels.

jorisvandenbossche

Apart from my last two comments, looks good!

pep8speaks · 2018-03-27T13:55:35Z

Hello @cbrnr! Thanks for updating the PR.

In the file pandas/io/formats/format.py, following are the PEP8 issues :

Line 648:80: E501 line too long (81 > 79 characters)

jorisvandenbossche · 2018-03-27T14:00:15Z

pandas/io/formats/format.py

@@ -625,7 +625,7 @@ def to_string(self):
                max_len += size_tr_col  # Need to make space for largest row
                # plus truncate dot col
                dif = max_len - self.w
-                adj_dif = dif
+                adj_dif = dif + 1  # see GH PR #17023


can you put it on the line above?

You mean

dif = max_len - self.w # see GH PR #17023 adj_dif = dif

?

dif is never used so we might as well skip it completely.

No, I just meant to put the comment on its own line, not on the same line after the code, like

# '+ 1' to avoid too wide repr (GH PR #17023) adj_dif = dif + 1

I'm sorry, of course, I'll change that.

jorisvandenbossche · 2018-03-28T07:51:56Z

@cbrnr Thanks a lot for this (and for your patience getting this merged :))

Change `max_columns` to `0` (automatically adapt the number of displayed columns to the actual terminal width)

TomAugspurger added the Output-Formatting __repr__ of pandas objects, to_string label Jul 19, 2017

humford mentioned this pull request Jul 19, 2017

"display.width" should default to 'None' #11515

Closed

jreback reviewed Jul 20, 2017

View reviewed changes

Remove + character

3cf51ca

Add images back to doc/source/_static

1eb3dad

Use correct image path

d01c682

jorisvandenbossche reviewed Mar 27, 2018

View reviewed changes

jorisvandenbossche approved these changes Mar 27, 2018

View reviewed changes

Include comment with GitHub PR number

308be12

Revert line

e77eb55

jorisvandenbossche reviewed Mar 27, 2018

View reviewed changes

cbrnr added 2 commits March 27, 2018 16:11

Put comment on separate line

ab63657

Remove unnecessary section anchor

f795914

jorisvandenbossche approved these changes Mar 27, 2018

View reviewed changes

jorisvandenbossche merged commit c9e8f59 into pandas-dev:master Mar 28, 2018

cbrnr deleted the nicer_display_defaults branch March 28, 2018 08:30

cbrnr mentioned this pull request Mar 28, 2018

Set pd.options.display.max_rows = 20 by default #20514

Closed

4 tasks

javadnoorb pushed a commit to javadnoorb/pandas that referenced this pull request Mar 29, 2018

Set pd.options.display.max_columns=0 by default (pandas-dev#17023)

72524e8

Change `max_columns` to `0` (automatically adapt the number of displayed columns to the actual terminal width)

dworvos pushed a commit to dworvos/pandas that referenced this pull request Apr 2, 2018

Set pd.options.display.max_columns=0 by default (pandas-dev#17023)

b28690b

Change `max_columns` to `0` (automatically adapt the number of displayed columns to the actual terminal width)

kornilova203 pushed a commit to kornilova203/pandas that referenced this pull request Apr 23, 2018

Set pd.options.display.max_columns=0 by default (pandas-dev#17023)

6f85061

Change `max_columns` to `0` (automatically adapt the number of displayed columns to the actual terminal width)

WillAyd mentioned this pull request Sep 2, 2018

BUG: Weird console formatting defaults #22524

Open

Set pd.options.display.max_columns=0 by default #17023

Set pd.options.display.max_columns=0 by default #17023

Conversation

cbrnr commented Jul 19, 2017 • edited Loading

TomAugspurger commented Jul 19, 2017

cbrnr commented Jul 19, 2017

chris-b1 commented Jul 19, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Jul 20, 2017

cbrnr commented Jul 20, 2017

jreback commented Jul 20, 2017

cbrnr commented Jul 20, 2017

cbrnr commented Jul 20, 2017 • edited Loading

chris-b1 commented Jul 20, 2017

takluyver commented Jul 20, 2017

cbrnr commented Jul 21, 2017

takluyver commented Jul 25, 2017

jreback commented Sep 23, 2017

cbrnr commented Sep 24, 2017

jreback commented Sep 24, 2017

cbrnr commented Oct 18, 2017

cbrnr commented Nov 7, 2017

jreback commented Nov 7, 2017

cbrnr commented Nov 9, 2017

jreback commented Nov 9, 2017

cbrnr commented Nov 9, 2017

jreback commented Nov 9, 2017

jreback commented Nov 9, 2017

cbrnr commented Nov 9, 2017

jreback commented Nov 9, 2017

codecov bot commented Nov 9, 2017 • edited Loading

Codecov Report

codecov bot commented Nov 9, 2017 • edited Loading

Codecov Report

jorisvandenbossche commented Mar 27, 2018 • edited Loading

cbrnr commented Mar 27, 2018

jorisvandenbossche commented Mar 27, 2018

jorisvandenbossche commented Mar 27, 2018

cbrnr commented Mar 27, 2018

cbrnr commented Mar 27, 2018

jorisvandenbossche commented Mar 27, 2018

cbrnr commented Mar 27, 2018

cbrnr commented Mar 27, 2018

jorisvandenbossche commented Mar 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorisvandenbossche left a comment

Choose a reason for hiding this comment

pep8speaks commented Mar 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorisvandenbossche commented Mar 28, 2018

cbrnr commented Jul 19, 2017 •

edited

Loading

cbrnr commented Jul 20, 2017 •

edited

Loading

codecov bot commented Nov 9, 2017 •

edited

Loading

codecov bot commented Nov 9, 2017 •

edited

Loading

jorisvandenbossche commented Mar 27, 2018 •

edited

Loading