Switch from using a homegrown logging solution to the stdlib logging module

[dstufft]

Overview

This PR switches us away from using our own custom logging framework and instead uses a combination of the stdlib logging and the warnings module. This will ideally give us a few benefits:

• Make it easier for debug logging to include information from other libraries that we're using.
• Ideally have less issues with unicode handling as we're offloading that to the standard library.
• Use a system that more Python developers will be familiar with to make it easier for end users to contribute to pip.
• Make pip friendlier in an eventual future where we might support using pip as a library instead of only as a CLI application.
• Easily add nice features to the logging, such as being able to add an appending log instead of an overwriting log that is on by default which automatically limits it's size.
• Less code for us to maintain!

Details

• Logger - When logging things in pip you'll now use a logger module that comes from logging.getLogger(__name__) instead of a global one at pip.log:logger.
• Logging Levels - We'll now have the standard logging levels, DEBUG, INFO, WARNING, ERROR, CRITICAL. By default we'll log INFO and higher, while any --quiet configuration will change that to ERROR and higher, and any --verbose will change that to DEBUG and higher. The difference between ERROR and CRITICAL is that CRITICAL should generally only be used for fatal errors.
• Deprecations - Instead of using logger.deprecated() deprecations will now be handled by using warnings.warn() with special warnings from the pip.utils.deprecation module.
• Backports logging.config.dictConfig from Python 2.7 for use in Python 2.6.

Tasks

• Switch from using pip.log:Logger to the stdlib logging module.
• Switch to using the warnings module to handle deprecations.
• Switch to using a logging formatter to handle indenting the log messages.
• Create a context manager to indent all the logging messages inside of the context manager.
• Simplify the --verbose and --quiet flags.
• Switch to using the stdlib logging modules to handle the log files that we write.
• Switch to using a custom log handler to colorize the output of the various levels.
• Use a UI module external to the logging system to handle progress indicators.
• Write tests for all of the new stuff and fix any broken tests.
• Refactor & Cleanup.

Fixed/Closed/Obsoleted Tickets

Closes #1943
Closes #1936
Closes #1772
Closes #1717
Closes #1629
Closes #1865
Closes #1826
Closes #1766
Closes #1713
Closes #1501
Closes #915

[dstufft]

Hey @pfmoore, could I bother you to pull this on Windows and make sure it doesn't break anything? I don't think it should but I'm having problems getting a Windows environment set up correctly.

[pfmoore]

@dstufft I grabbed the branch and did some basic tests (install, uninstall, search) and provoked a few errors (package not found, only external versions available). No problems with any of that.

One point. The "Some externally hosted files were ignored as access to them may be unreliable" message is no longer shown with -q where it was previously. Also there's no debug log message. That doesn't seem like it'd be Windows-specific, so was that deliberate?

New version:

>pip install -q cdecimal
  Could not find any downloads that satisfy the requirement cdecimal
  No distributions at all found for cdecimal

current develop version:

>pip install -q cdecimal
  Could not find any downloads that satisfy the requirement cdecimal
  Some externally hosted files were ignored as access to them may be unreliable (use --allow-external cdecimal to allow).
No distributions at all found for cdecimal
Storing debug log for failure in C:\Users\Gustav\pip\pip.log

[pfmoore]

If there's anything else specifically you'd like me to test, let me know.

[dstufft]

Ah yes, the cdecimal thing was because I wasn't sure if -q should show WARNINGS+ or ERROR+. I removed the cumulative nature of -q and -v, one does as many as three does now. This simplifies are level information.

I guess -q should be WARNINGS+ instead of ERRORS+, I'll make that change now.

[dstufft]

And no, nothing specific, I just wanted to make sure I didn't break anything on Windows. I didn't think there would be any changes that would break anything there.

[dstufft]

Oh, and the debug log message, maybe I should still print that message hmm. Historically the old debug log was overwritten with the entire log stream anytime the pip command failed. I've changed that in this PR so that it's always written, no matter what and it will rotate itself once it gets 10MB in size. So now there is always a log and it's not just special to failures.

[pfmoore]

The log bit seems fine, I probably should have read the doc changes ;-)

[Ivoz]

If #2012 ends up looking good, I'd presume the easiest course of action might be to merge that before this and then rebase this on top of it, as it touches compat.py.

[dstufft]

Technically this obsoletes #2012, but 2012 has tests that I haven't written yet so I'm fine with that.