Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix leak in _PyCode_Update #48

Merged
merged 6 commits into from
Jul 26, 2021
Merged

fix leak in _PyCode_Update #48

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
cdc36c5
fix leak in _PyCode_Update
iritkatriel Jul 23, 2021
038c4da
init name fields in _PyCode_New
iritkatriel Jul 24, 2021
daf9729
bump magic number (fixes crashes in test_importlib and test_scope)
iritkatriel Jul 24, 2021
0bc1f4c
add question
iritkatriel Jul 24, 2021
f44cd7c
Revert "add question"
iritkatriel Jul 25, 2021
5bacfe7
move the co_*name decrefs to _PyCode_Update
iritkatriel Jul 25, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Lib/importlib/_bootstrap_external.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -372,7 +372,7 @@ def _write_atomic(path, data, mode=0o666):
# Whenever MAGIC_NUMBER is changed, the ranges in the magic_values array
# in PC/launcher.c must also be updated.

MAGIC_NUMBER = (9994).to_bytes(2, 'little') + b'\r\n'
MAGIC_NUMBER = (9996).to_bytes(2, 'little') + b'\r\n'
_RAW_MAGIC_NUMBER = int.from_bytes(MAGIC_NUMBER, 'little') # For import.c

_PYCACHE = '__pycache__'
Expand Down
11 changes: 10 additions & 1 deletion Objects/codeobject.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -324,6 +324,7 @@ init_code(PyCodeObject *co, struct _PyCodeConstructor *con)
co->co_name = con->name;
Py_INCREF(con->qualname);
co->co_qualname = con->qualname;

co->co_flags = con->flags;

Py_XINCREF(con->code);
Expand Down Expand Up @@ -420,6 +421,10 @@ _PyCode_New(struct _PyCodeConstructor *con)
PyErr_NoMemory();
return NULL;
}
co->co_filename = NULL;
co->co_name = NULL;
co->co_qualname = NULL;

Comment on lines +424 to +427
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, I changed my mind again. I think we should leave init_code() unchanged (storing values without looking) and in _PyCode_Update() we explicitly DECREF co_name, co_qualname and co_filename (because on entering there we know they are valid).

Moreover, in _PyCode_Update() we should actually preserve the original co_filename, and if it's different, overwrite the co_filename of any (newly created) dehydrated code objects, probably using the same logic as update_compiled_module() in import.c. In case you wonder why, the reason is to preserve the filename set at the toplevel by _imp.fix_co_filename() under some circumstances (called from _bootstrap_external.py) -- there's an obscure test in Lib/test/test_import/init.py (test_incorrect_code_name) that comes down to verifying that if a module is loaded from bytecode, the co_filename of all the loaded code objects is where it's loaded from, not whatever was marshaled into the file. The existing logic doesn't set this for code objects that are skipped by lazy loading (I already had to fix a crash there due to co_consts being NULL), and hydrating currently overwrites the fixed co_filename with the incorrect one read from the marshal data. (Arguably we should not serialize co_filename at all, and only set it when loading. But that's probably going to break some other test...)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I saw that test but didn't understand it. I'll try this in a new PR. I think update_code_filenames needs to move from import.c to codeobject.c.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I spent some quality time with both C and Python debuggers before I understood the feature that it tests. :/ The key is the _imp.fix… call in _bootstrap_external.py.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See iritkatriel#21

init_code(co, con);

return co;
Expand All @@ -442,7 +447,11 @@ _PyCode_Update(struct _PyCodeConstructor *con, PyCodeObject *code)
con->columntable = Py_None;
}

init_code(code, con); // TODO: This leaks!
Py_XDECREF(code->co_filename);
Py_XDECREF(code->co_name);
Py_XDECREF(code->co_qualname);

init_code(code, con);

return code;
}
Expand Down
49 changes: 24 additions & 25 deletions Programs/test_frozenmain.h
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Python/importlib_external.h
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.