Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Postpone pandoc execution to the end of all processing #1709

Closed
oesteban opened this issue Jul 20, 2019 · 0 comments · Fixed by #1710
Closed

Postpone pandoc execution to the end of all processing #1709

oesteban opened this issue Jul 20, 2019 · 0 comments · Fixed by #1710
Assignees
@oesteban
Labels
bug effort: low Estimated low effort task impact: high Estimated high impact task

Comments

@oesteban
Copy link
Member

Soichi found that the memory peak at the beginning of fMRIPrep is caused by pandoc, which is aligned to this open issue in pandoc.

Since that issue has been open for a while now, but we are not going to think of an alternative just for the conversion of markdown to html to generate the boilerplate, a good solution is to postpone pandoc to be the latest process run, and also offer a flag to totally skip the conversion.
@oesteban oesteban added bug impact: high Estimated high impact task effort: low Estimated low effort task labels Jul 20, 2019
@oesteban oesteban self-assigned this Jul 20, 2019
oesteban added a commit to oesteban/fmriprep that referenced this issue Jul 21, 2019
@oesteban
FIX: Postpone pandoc conversion of boilerplate after workflow has…
244f0d4 
… fully run.

This PR addresses the memory peak caused by pandoc, running the process
at the very end of fMRIPrep.

We probably want to explicitly clear up as many objects as we can from
memory and force gc before the subprocess opens pandoc, but that's left
for a future PR.

This PR just closes nipreps#1709 with the delayed pandoc run. I'm not
implementing the flag to avoid the boilerplate generation because I'm
not sure we want to introduce an additioinal argument for this.

Coincidentally, it also closes nipreps#1667 - now the deletion of citation
files is done inside a try..catch where ``FileNotFoundError`` is
dismissed. That should avert the issue, while keeping the functionality
of nipreps#1567.
@oesteban oesteban mentioned this issue Jul 21, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@oesteban oesteban

Labels
bug effort: low Estimated low effort task impact: high Estimated high impact task
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

FIX: Postpone pandoc conversion of boilerplate after workflow has fully run oesteban/fmriprep
1 participant
@oesteban