DISCO methods paper, initial attempt to address feedback from reviewers 2&3 #1033
Conversation
|
AppVeyor build 1.0.4445 for commit 9a38dcd is now complete. Found 8 potential spelling error(s). Preview:content/22.vaccines.md:21:devleopmentcontent/22.vaccines.md:74:appraoches content/22.vaccines.md:127:passaging content/23.vaccines-app.md:15:IgGs content/23.vaccines-app.md:387:IgGs content/60.methods.md:45:exessive content/60.methods.md:105:MAARIE content/60.methods.md:250:Manubut... |
There was a problem hiding this comment.
Thanks for starting on these updates @rando2. Because we only have a few days to finish these revisions, should we work together in this pull request to make the remaining edits?
I know this is still a work in progress, but I added some initial thoughts and text you can merge in.
content/60.methods.md
Outdated
| However, the barriers to entry posed by git and GitHub likely still reduced participation from individuals who might have otherwise been interested. | ||
| Additionally, using an approach that hinged on these tools likely biased our contributors towards those who were interested in or experienced with computational tools. | ||
| The trajectory of the pandemic itself also likely influenced participation: engagement waned over the course of the pandemic as labs opened back up and researchers were able to return to their work, and we recruited very few medical doctors to the project, which is unsurprising given the load on medical professionals during this time. | ||
| These are limitations that we hope to improve through future work on Manubot. |
There was a problem hiding this comment.
I'm not sure that these are limitations that we can improve through Manubot. It is possible that this is the nature of a large open collaborative project. Many contributors are excited initially and participate in their areas of expertise. A much smaller subset sustain the project to complete.
We could refer to the deep review trends and compare them with contribution trends here. See https://greenelab.github.io/meta-review/#fig:contrib There are many authors who only have their initial contributions in early or mid 2017.
I think it is fair to acknowledge that as a downside of this type of approach. It would be stronger if we can think of other (non-Manubot) collaborative projects, whether writing or software development or something else, to generalize contributor trends.
Edit: never mind, I see you have a paragraph about this below already. That's the idea I was going for.
There was a problem hiding this comment.
I think I need to move line 282 to the paragraph below to make this more clear, we cannot fix the pandemic with Manubot!
There was a problem hiding this comment.
Just wanted to remark that I'm in the process of reorganizing these paragraphs and adding ideas from the discussion here, and I'll push the changes later this morning!
content/60.methods.md
Outdated
| This review was published on July 10, 2020, four days before Moderna released the surprisingly promising results of their phase 1 trial [@doi:10.1056/NEJMoa2022483] that changed expectations surrounding vaccines. | ||
| Therefore, the COVID-19 publishing climate presented a challenge where curation of the literature by a diverse group of experts in a format that could respond quickly to high-volume, high-velocity information was desirable. | ||
|
|
||
| We therefore sought to develop a platform for scientific discussion and collaboration around COVID-19 by adapting open publishing infrastructure to accommodate the scale of the COVID-19 publishing boom. | ||
| Recent advances in open publishing have created an infrastructure that facilitates distributed, version-controlled collaboration on manuscripts [@doi:10.1371/journal.pcbi.1007128].<!--To Do: possibly cite some other efforts here--> | ||
| Recent advances in open publishing have created an infrastructure that facilitates distributed, version-controlled collaboration on manuscripts [@doi:10.1371/journal.pcbi.1007128].<!--To Do: cite other efforts here & explain why Manubot was the best fit--> |
There was a problem hiding this comment.
Some initial text you can work from. Maybe this would fit as a new paragraph following this one.
We selected Manubot because it offers several advantages over comparable collaborative writing platforms such as Authorea, Overleaf, Google Docs, Word Online, or wikis [@doi:10.1371/journal.pcbi.1007128].
Citation-by-identifier ensures consistent reference metadata standards that would be difficult to maintain manually in a manuscript with dozens of authors and over 1,500 citations.
Manubot's pull request-based contribution model balances the goals of making the project open to everyone and maintaining scientific accuracy.
All contributions are reviewed, discussed, and formally approved on GitHub before text updates appear in the public-facing manuscript at https://greenelab.github.io/covid19-review/.
Continuous integration (CI) seamlessly combines author-produced text and figures with automatically generated and updated statistics and figures from external data sources and the manuscript's own content.
In addition, the authors who initially launched this project included Manubot developers who had prior successes using Manubot for massively open and traditional manuscript.
We could use analyze-ms-stats/manuscript_stats.json from #1019 to automate the exact number of references. The last sentence is meant to acknowledge that we also selected Manubot in part because of familiarity and our existing bias toward it (driven by prior successes).
There was a problem hiding this comment.
I need to get that PR finalized -- will work on that next!
There was a problem hiding this comment.
Once #1034 merges, we should be able to make all of these dynamic. There are several places throughout the text where they come up.
content/60.methods.md
Outdated
| Finally, the preliminary addition of LaTeX output is useful for researchers from computational fields who submit papers in TeX format and removes the step of reformatting markdown prior to submission. | ||
|
|
||
| <!-- To Do: Add table listing additions to manubot--> |
There was a problem hiding this comment.
Here is an initial summary:
- CI scripts to regularly download external data sources, generate figures and statistics, and store them in a GitHub branch that is read when Manubot builds the latest manuscript
- Zotero extension to report more relevant clinical trial metadata from https://clinicaltrials.gov
- Manubot feature to support citing any Compact Uniform Resource Identifier, such as
clinicaltrialsorncbigene - Integration with scite to track retractions, corrections, and notices of concern
- CI spell-checking reports that post as pull request comments
- Improved support for Pandoc's LaTeX output
- Building individual sections of a larger manuscript as standalone documents alongside the complete manuscript
This feels redundant with the existing text. Do we agree with the reviewer that the summary is helpful?
There was a problem hiding this comment.
Yours is better than mine! I do like the idea of a table, because I have to keep looking through the text to jog my memory about the full list!
| As a result, the lead authors of each paper often spent several weeks prior to submission updating the text to reflect new developments in each area. | ||
| In the future, it may be possible to streamline this process through integration with a tool such as CoronaCentral [@doi:10.1073/pnas.2100766118] to automatically identify relevant, high-impact papers that need to be included, although expertise would still be required to incorporate them. | ||
| Such a functionality would serve to address another major limitation to this project: ultimately, the scale of the infodemic made it incredibly challenging to stay on top of the COVID-19 literature, even with engagement from a large number of collaborators from diverse backgrounds. | ||
|
|
There was a problem hiding this comment.
Let's consider additional limitations of the Manubot approach. Ryan brought up a good one at #947 (review) Readers of the HTML manuscript cannot tell what parts of the paper are up to date and which are stale. A savvy reader could use GitHub's blame feature to determine when each sentence or figure was last edited, but that information is not displayed in the rendered manuscript and doesn't distinguish between typo corrections and substantive updates.
A related idea is that we have automating figures and statistics that are summarized by static contributed text. There is a risk that the data in the figure or statistics can change and the related text will be wrong or inconsistent. We sometimes tried to work around this by writing "As of date XYZ" in the text, but the core limitation remains.
The pull request model can sometimes be tedious. Adding even a single sentence requires making a pull request, waiting for an approval, and waiting for the CI build to finish. That has deterred some colleagues from joining Manubot projects in the past.
There is no good way to provide general comments on a manuscript. Comments can be added to pull requests, but only to the modified content. Hypothesis comments can be added anywhere but are not easy to integrate back into the manuscript like pull request comments or suggestions.
Any other limitations we've faced?
There was a problem hiding this comment.
Another challenge is definitely that we haven't pivoted completely away from DOCX. I have been manually adding commits that people send me either in emails or in docx attachments (using --author).
Also, there are silly issues like Casey's setting automatically remove EOL whitespace, so every time he opens a PR, it has a huge impact in terms of line numbers. We can fix this by automating this for every PR, but I haven't gotten to it!
Another one-- we don't have a good way to link up preprints with their published papers. I sometimes do this randomly while checking the text -- I'll open a preprint to confirm that I'm not changing the meaning with some edit, and then find that it has been published. I think we can also add this thanks to work David Nicholson has been doing.
There was a problem hiding this comment.
Other limitations from the reviews we could mention:
- We integrated with the Mount Sinai preprint reviews but any other relevant efforts (e.g. preLights). There are other potential contributors who are/were reviewing COVID-19 preprints. How could we build upon their existing efforts?
- scite slows down HTML load times (HTML load time #1015). Could we load the badges only on request?
- Formal user study of Manubot versus other collaborative writing approaches, provide evidence as to how usable (or not) Manubot is for non-technical users. Bias among current authors toward technical users (related to the point above about DOCX).
- Machine readable/actionable summaries of key information in the review manuscripts as a way to combat the infodemic (as opposed to creating yet another set of manuscripts to read)
There was a problem hiding this comment.
I believe these are all addressed in the new draft text!
|
AppVeyor build 1.0.4453 for commit 663849d is now complete. Found 9 potential spelling error(s). Preview:content/22.vaccines.md:21:devleopmentcontent/22.vaccines.md:74:appraoches content/23.vaccines-app.md:15:IgGs content/23.vaccines-app.md:387:IgGs content/60.methods.md:45:exessive content/60.methods.md:105:MAARIE content/60.methods.md:249:addons content/60.methods.md:249:Manubut content/60.methods.md:270:addons... |
Co-authored-by: Anthony Gitter <agitter@users.noreply.github.com>
|
AppVeyor build 1.0.4455 for commit ee85f29 is now complete. Found 9 potential spelling error(s). Preview:content/22.vaccines.md:21:devleopmentcontent/22.vaccines.md:74:appraoches content/23.vaccines-app.md:15:IgGs content/23.vaccines-app.md:387:IgGs content/60.methods.md:45:exessive content/60.methods.md:105:MAARIE content/60.methods.md:249:addons content/60.methods.md:249:Manubut content/60.methods.md:270:addons... |
|
AppVeyor build 1.0.4460 for commit 9b37b7f is now complete. Found 10 potential spelling error(s). Preview:content/22.vaccines.md:21:devleopmentcontent/22.vaccines.md:74:appraoches content/23.vaccines-app.md:15:IgGs content/23.vaccines-app.md:387:IgGs content/60.methods.md:45:exessive content/60.methods.md:105:MAARIE content/60.methods.md:249:addons content/60.methods.md:249:Manubut content/60.methods.md:265:ncbigene content/60.methods.md:272:addons... |
|
AppVeyor build 1.0.4462 for commit 6ffeeda is now complete. Found 10 potential spelling error(s). Preview:content/22.vaccines.md:21:devleopmentcontent/22.vaccines.md:74:appraoches content/23.vaccines-app.md:15:IgGs content/23.vaccines-app.md:387:IgGs content/60.methods.md:45:exessive content/60.methods.md:106:MAARIE content/60.methods.md:252:addons content/60.methods.md:252:Manubut content/60.methods.md:268:ncbigene content/60.methods.md:275:addons... |
content/60.methods.md
Outdated
| However, the barriers to entry posed by git and GitHub likely still reduced participation from individuals who might have otherwise been interested. | ||
| Additionally, using an approach that hinged on these tools likely biased our contributors towards those who were interested in or experienced with computational tools. | ||
| The trajectory of the pandemic itself also likely influenced participation: engagement waned over the course of the pandemic as labs opened back up and researchers were able to return to their work, and we recruited very few medical doctors to the project, which is unsurprising given the load on medical professionals during this time. | ||
| These are limitations that we hope to improve through future work on Manubot. |
There was a problem hiding this comment.
Just wanted to remark that I'm in the process of reorganizing these paragraphs and adding ideas from the discussion here, and I'll push the changes later this morning!
|
Yesterday I pushed two commits to work on the figures and speed up the build time by disabling the pharmaceuticals docx build. The footnotes look pretty good on the HTML and PDF. However, we should be consistent about the order when we have a footnote and a citation. We'll also need to test the footnotes in LaTeX. |
Co-authored-by: HM Rando <halie.rando@cuanschutz.edu>
I think footnote before citation is the better of the two options, like 3 and 4 in the screenshot above. Otherwise it looks like the footnote is modifying the citation. |
|
AppVeyor build 1.0.4469 for commit 6dc2c9e is now complete. Found 11 potential spelling error(s). Preview:content/22.vaccines.md:21:devleopmentcontent/22.vaccines.md:74:appraoches content/23.vaccines-app.md:15:IgGs content/23.vaccines-app.md:387:IgGs content/60.methods.md:77:Authorea content/60.methods.md:258:addons content/60.methods.md:262:useres content/60.methods.md:283:addons content/60.methods.md:295:PRs content/60.methods.md:306:ansynchrony content/60.methods.md:307:colloration... |
|
AppVeyor build 1.0.4472 for commit a962215 is now complete. Found 9 potential spelling error(s). Preview:content/22.vaccines.md:21:devleopmentcontent/22.vaccines.md:74:appraoches content/23.vaccines-app.md:15:IgGs content/23.vaccines-app.md:387:IgGs content/60.methods.md:77:Authorea content/60.methods.md:249:addons content/60.methods.md:274:addons content/60.methods.md:286:PRs content/60.methods.md:297:asynchrony... |
|
AppVeyor build 1.0.4484 for commit fd2186f is now complete. Found 6 potential spelling error(s). Preview:content/22.vaccines.md:21:devleopmentcontent/22.vaccines.md:74:appraoches content/23.vaccines-app.md:15:IgGs content/23.vaccines-app.md:387:IgGs content/60.methods.md:297:asynchronicity content/60.methods.md:306:inclusivity... |
| This computational approach allows for some of the updating process to be off-loaded so that domain experts can focus on the broader implications of new information as it emerges. | ||
| As a result, centralizing, summarizing, and critiquing data and literature broadly relevant to COVID-19 can help to expedite the interdisciplinary scientific process that is currently happening at an advanced pace. | ||
| As of September 13, 2021, almost 3,000 commits have been made to the manuscript across 575 merged pull requests. |
There was a problem hiding this comment.
Woah! That's a lot.
Where did the 3,000 commits come from? Is this one of the stats you now track in a JSON file? It looks like it distinguishes between commits associated with pull requests versus all 5,000 commits to master (which includes rootstock and non-content commits).
There was a problem hiding this comment.
Never mind, now I see "manuscript_commits": 2886 in covid19-review-stats.json
[ci skip] This build is based on 5611bd9. This commit was created by the following CI build and job: https://github.com/greenelab/covid19-review/commit/5611bd91a215753f643fadfa483ab07c3895f314/checks https://github.com/greenelab/covid19-review/runs/1231893101
[ci skip] This build is based on 5611bd9. This commit was created by the following CI build and job: https://github.com/greenelab/covid19-review/commit/5611bd91a215753f643fadfa483ab07c3895f314/checks https://github.com/greenelab/covid19-review/runs/1231893101
Description of the proposed additions or changes
This is a draft, I need to finish up these responses tomorrow.
Related issues
Suggested reviewers (optional)
@agitter tagging you just to loop you in that responses to most of the comments are in the works. It's not ready for review quite yet though.
Checklist