ENH: Automated Bug Reporting #92

teddyrendahl · 2018-03-27T00:37:17Z

Description

The idea of this PR is that user should be able to report a bug from the IPython session that will eventually make its way to Bug-Reports. Because we have standardized the way that we organize logs, CONDA environments e.t.c we can tell a lot about the current Python environment by just poking the right environment variables e.t.c.. To see the polished example of what the bug report will look like checkout https://github.com/teddyrendahl/Bug-Reports/issues/1. The process is kind of multi-step so I figure I'd give a quick walkthrough here. Everything is kicked off when the user call the report_bug function. This can either be used as an IPython magic to report a bug in a single line of code, or run as a regular function where you can grab the last n commands from the shell.

#IPython Magic
%report_bug my_buggy_function(foo=bar)
# Standard function call
foo = 1
bar = 2
first_buggy_function(foo)
second_buggy_function(bar)
report_bug(prior_commands=4)

We then ask the operator some brief questions about the issue we will not be able to determine programatically.

Please provide a one sentence description of the problem you are encountering: Issue running fake scan
How many of the previous commands are relevant  to the issue you would like investigated?:  5
Please enter a name so we can follow-up with additional questions: trendahl

A vim window with a little prompt is also opened up so that the operator can type a more detailed explanation of the issue that looks like

# Please describe the issue you are wishing to report. What did you expect to
# happen? What actually happened? Does this issue occur every time you use this
# function?

# Lines that start with a '#' will be ignored. Save the session and exit to
# store the description. Press "i" to be begin typing, then "Esc" followed by
# ":wq" to exit.

Now the major issue is that we can't expect every operator to have a GitHub account so instead when the issue is created it gets stored in a JSON file in a known location in NFS. Once this is ready for ops we can setup a CRON job with a reasonable frequency to scan the directory, find new issues and post them to GitHub. This will probably be done with a pcdsbot account I'll setup. The post_to_github function is there when this is ready to go, and I also created the full script as hutch-python/bin/post-issues

Motivation and Context

Closes #64

How Has This Been Tested?

Tests added for relevant top-level functions.

Where Has This Been Documented?

Added this to the Useful Utilities page of the User Documentation. Also updated the Internal API Reference

Instead of repeating the code to find a handler of a certain name, a generic function was created to find a handler of any name. This is then used to find the base log file name and search the directory for any other log messages that may have been created.

codecov-io · 2018-03-27T00:46:42Z

Codecov Report

Merging #92 into master will decrease coverage by 1.12%.
The diff coverage is 92.24%.

@@            Coverage Diff             @@
##           master      #92      +/-   ##
==========================================
- Coverage   98.75%   97.62%   -1.13%     
==========================================
  Files          14       15       +1     
  Lines         560      674     +114     
==========================================
+ Hits          553      658     +105     
- Misses          7       16       +9

Impacted Files	Coverage Δ
hutch_python/cli.py	`100% <100%> (ø)`	⬆️
hutch_python/constants.py	`100% <100%> (ø)`	⬆️
hutch_python/log_setup.py	`100% <100%> (ø)`	⬆️
hutch_python/bug.py	`90.81% <90.81%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4ab1d18...56d7601. Read the comment docs.

teddyrendahl · 2018-03-27T16:28:43Z

After sleeping on it. I think I'll switch this to use an OAuth token and post issues immediately instead of the folder and CRON job. Recommending that this is done in a follow-up PR though.

ZLLentz

I like this implementation, see comments for suggestions for changes. Consider trying to bump up the coverage too, bug.py is the lowest-coverage file in the module by a fair bit.

ZLLentz · 2018-03-27T16:12:17Z

bin/post-issues

+
+
+description = """\
+Post all of the stored issue to https://github.com/pcdshub/Bug-Reports


typo, issue should be issues

ZLLentz · 2018-03-27T16:16:36Z

bin/post-issues

+    pw = args.password or getpass.getpass()
+    # Run post_to_github on every stored issue
+    for issue in os.listdir(BUG_REPORT_PATH):
+        if issue.endswith('.json'):


Is anything else in this folder? It seems odd to me to check for the file extension when that isn't sufficient to guarantee that we have a well-formed bug report. I guess it's a good first cut though if someone mangles the BUG_REPORT_PATH

ZLLentz · 2018-03-27T16:18:11Z

bin/post-issues

+                                     formatter_class=fclass)
+    parser.add_argument('-u', '--user', dest='username',
+                        default=None,
+                        help="Username for GitHub account")


It looks like this is an optional argument, which is odd because we don't prompt the user for it if omitted. I expect the script breaks if this isn't passed.

ZLLentz · 2018-03-27T16:22:14Z

docs/source/user_utils.rst

+without valid GitHub accounts. Once issues are received on GitHub the
+appropriate action will be made by the PCDS staff. This may mean a deeper look
+at the linked log files and/or creating a distilled issue or action item in a
+different repository. 


None of the bug reporting stuff belongs on this page, it's not a user utility that you'd include in a beamline file. It deserves it's own page before this one on the index.

Didn't realize this was only for beamline.py utilities. Might suggest a rename if this was your intention.

I agree, the page should be renamed. Even if it would fit on the page, bug reporting still deserves it's own page because of how important it is.

Makes sense. Bug Reporting page has been created 👍

ZLLentz · 2018-03-27T16:22:57Z

hutch_python/bug.py

+"""
+One advantage of standardizing the logging system and startup scripts with
+hutch-python is programatically being able to gather information about the
+currrent Python environment. One common use case for much of this information


currrent typo

ZLLentz · 2018-03-27T17:48:27Z

hutch_python/bug.py

+    # If the user somehow has no IPython history this is raised. A very rare
+    # occurence except for in Continuous Integration tests
+    except OSError:
+        logger.exception


What does this do without a call or inputs?

ZLLentz · 2018-03-27T17:49:42Z

hutch_python/bug.py

+    try:
+        logfiles = get_session_logfiles()
+    except RuntimeError:
+        logger.warning("No debug RotatingFileHandler configured for session")


Surely not every RuntimeError is because the rotating file handler is missing? I feel like get_session_logfiles() should return an empty list instead of raising an error when there are no session logfiles.

This was a hold over from the get_console_handler functionality.

ZLLentz · 2018-03-27T17:51:37Z

hutch_python/bug.py

+    @line_magic
+    def report_bug(self, line):
+        """Creates a bug_report while running the given line"""
+        # Enter both the DEBUG context and store the output of our command


Is this comment true?

ZLLentz · 2018-03-27T18:00:31Z

hutch_python/tests/test_bug.py

+        return r
+
+
+def test_bug_report(monkeypatch):


didn't realize this monkeypatch fixture existed, neat trick

ZLLentz · 2018-03-27T18:12:19Z

bin/post-issues

+                logger.exception("Error posting %s", issue)
+
+
+if __name__ == '__main__':


Can you add a test that runs this script? Look for my tests that shell out and call hutch-python. Alternatively, move main into bug.py and test it there.

teddyrendahl · 2018-03-28T21:45:07Z

This is ready for review again.

ZLLentz · 2018-03-28T22:08:26Z

hutch_python/bug.py

+    path = os.path.join(BUG_REPORT_PATH, '{}.json'.format(str(uuid.uuid4())))
+    logger.info("Saving JSON representation of bug report %s", path)
+    simplejson.dump(report, open(path, 'w+'))
+    return path


Did you change your mind about making this skip the save file step?

Never mind, you wanted to do this in a separate PR.

ZLLentz · 2018-03-28T22:09:47Z

I'm still uneasy about the low coverage. If you'd like, I'll suggest lines that should be covered.

teddyrendahl · 2018-03-28T22:35:00Z

Fair retort. The major hole is the get_text_from_editor function because I was too lazy to mock a vim session. I'll write this up.

teddyrendahl · 2018-03-29T00:37:45Z

Added two additional tests for the specifics of get_current_environment and get_text_from_editor. Uncovered an issue if PYTHONPATH is configured but not pointing to a directory that exists so 👏 for being persistent @ZLLentz

I left the BugMagics class untested. Not very clear how to do it and the function is largely just making IPython API calls we assume are correct then calling report_bug which is tested already.

ZLLentz

👍

teddyrendahl added 19 commits March 24, 2018 11:58

TST: Add test to find log files from test session

bc9d60b

TST: Add a test for generating bug report

9ee76d1

DEV: Github issue Jinja2 template

9ff2c53

MAINT: Keep internal variable for report directory

ad2aa12

BLD: Explicit dependencies on requests, simplejson and jinja2

b98dedc

ENH: JSON report saving and GitHub posting

335d145

TST: Test GitHub posting with fake session

257afd0

FIX: Rejoin textwrap from editor

6513714

STY: Various PEP8 violations

8810353

BLD: Include jinja2 template in package

f487ed2

MAINT: Try / except block for no IPython history

29ffd9c

ENH: Create report_bug magics

d3d555f

MAINT: Register report_bug magics in IPython shell

47fd7ce

ENH: post-issues command line function

9e7f59d

DOC: Full API documentation and module explanation

b83111a

DOC: Add get_session_logfiles to API

6175dbf

DOC: Break up utils into sections, include bug reporting

bd8da30

DOC: Full API of bug.py available in autosummary

01ecf5f

teddyrendahl requested a review from ZLLentz March 27, 2018 00:37

MAINT: Switch to official Bug-Reports repository

b926680

ZLLentz requested changes Mar 27, 2018

View reviewed changes

teddyrendahl added 6 commits March 27, 2018 11:30

REV: Remove CRON job post-issues script

0adc78c

DOC: Various typos and formatting issues

7c9ff14

REV: Remove bug report information for Useful Utilities

5b38028

DOC: Reorganize placement of various docstrings

6d805e0

DOC: Remove references to CRON job

5f72c03

FIX: Forgotten description in exception logging

6113df5

teddyrendahl added 4 commits March 27, 2018 11:58

MAINT: Clarify various comments

1309b0d

MAINT: Check EDITOR environment variable instead of assuming vim

666bc8a

MAINT: Place try/except block in lower level get_session_logfiles

4f59dab

STY: One blank line instead of two between imports and logger init

f5c0113

ZLLentz reviewed Mar 28, 2018

View reviewed changes

teddyrendahl added 3 commits March 28, 2018 16:01

TST: Add explicit test for dev_pkgs and CONDA env variables

ac0715e

FIX: Return empty list even if PYTHONPATH is not found

83d0ace

TST: Add mock test for get_text_from_editor

56d7601

ZLLentz approved these changes Mar 29, 2018

View reviewed changes

teddyrendahl merged commit ac9d8f7 into pcdshub:master Mar 29, 2018

teddyrendahl deleted the bug_report branch March 29, 2018 20:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: Automated Bug Reporting #92

ENH: Automated Bug Reporting #92

teddyrendahl commented Mar 27, 2018

codecov-io commented Mar 27, 2018 •

edited

Loading

teddyrendahl commented Mar 27, 2018

ZLLentz left a comment

ZLLentz Mar 27, 2018

ZLLentz Mar 27, 2018

ZLLentz Mar 27, 2018

ZLLentz Mar 27, 2018

teddyrendahl Mar 27, 2018

ZLLentz Mar 27, 2018

teddyrendahl Mar 27, 2018

ZLLentz Mar 27, 2018

ZLLentz Mar 27, 2018

ZLLentz Mar 27, 2018

teddyrendahl Mar 27, 2018

ZLLentz Mar 27, 2018

ZLLentz Mar 27, 2018

ZLLentz Mar 27, 2018

teddyrendahl commented Mar 28, 2018

ZLLentz Mar 28, 2018

ZLLentz Mar 28, 2018

ZLLentz commented Mar 28, 2018

teddyrendahl commented Mar 28, 2018

teddyrendahl commented Mar 29, 2018

ZLLentz left a comment



		description = """\
		Post all of the stored issue to https://github.com/pcdshub/Bug-Reports

		logger.exception("Error posting %s", issue)


		if __name__ == '__main__':

ENH: Automated Bug Reporting #92

ENH: Automated Bug Reporting #92

Conversation

teddyrendahl commented Mar 27, 2018

Description

Motivation and Context

How Has This Been Tested?

Where Has This Been Documented?

codecov-io commented Mar 27, 2018 • edited Loading

Codecov Report

teddyrendahl commented Mar 27, 2018

ZLLentz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

teddyrendahl commented Mar 28, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ZLLentz commented Mar 28, 2018

teddyrendahl commented Mar 28, 2018

teddyrendahl commented Mar 29, 2018

ZLLentz left a comment

Choose a reason for hiding this comment

codecov-io commented Mar 27, 2018 •

edited

Loading