update methods part 1 (collaboration)

content/60.methods.md

@@ -70,26 +70,21 @@ By recording the evolution of information over time and assembling a resource th
 
 ### METHODS
 
-#### Contributor Recruitment
+#### Contributor Recruitment and Roles
 
-A preliminary requirement for this undertaking was to establish Manubot as a platform accessible to researchers with limited computational training, as is common in biology and medicine, where version control has historically not been emphasized [@doi:10.1109/SE4Science.2017.11].
-Given the limitations imposed upon scientists by the COVID-19 pandemic and social distancing measures that had most scientists (including students) working from home for much of 2020, community building across disciplines and across career stages was a priority of the project.
-The current project was managed through GitHub [@url:https://github.com/greenelab/covid19-review] using Manubot [@doi:10.1371/journal.pcbi.1007128] to continuously generate a version of the manuscript online [@url:https://greenelab.github.io/covid19-review].
+A preliminary requirement for this undertaking was to establish Manubot as a platform accessible to researchers with limited experience working with git, as is common in biology and medicine, where version control is not typically emphasized [@doi:10.1109/SE4Science.2017.11; @doi:10.1177/2515245918754826; @doi:10.1186/1751-0473-8-7].
 Contributors were recruited by word of mouth and on Twitter, and we sought out opportunities to integrate existing efforts to train early-career researchers (ECRs).
-Few researchers in biological and medical fields are trained in version control tools such as git <!--To Do: Find some sort of reference for this, software carpentry? Pull from citations to 10.1371/journal.pcbi.1004668
--->
-
-In order to make the project accessible to individuals from a number of backgrounds, we developed resources explaining how to use GitHub's web interface to develop and edit text and interact with Manubot for individuals with no prior experience working with git or other version control platforms.
-<!--To Do: elaborate on resource development-->
-
-Interested parties were encouraged to contribute in a number of ways.
-One option was to submit articles of interest as issues in the GitHub repository.
-Articles were classified as _diagnostic_, _therapeutic_, or _other_, and a template was developed to guide the review of papers and preprints in each category.
-Following a framework often used for assessing medical literature, the review consisted of examining methods used in each relevant article, assignment (whether the study was observational or randomized), assessment, results, interpretation, and how well the study extrapolates [@doi:10.5014/ajot.60.4.367].
-For examples of each template, please see Appendices B-D.
-Another option was to contribute or edit text using GitHub's pull request system.
-Each pull request was reviewed and approved by at least one other author.
-Manubot also provides a functionality to create a bibliography using digital object identifiers (DOIs), website URLs, or other identifiers such as PubMed identifiers and arXiv IDs.
+We invited potential collaborators to contribute a short introduction on a GitHub issue in order to collect information about who was involved and provide an introduction to working with GitHub issues.
+Interested participants were encouraged to contribute in a number of ways.
+One option was to catalog articles of interest as issues in the GitHub repository.
+We developed a standardized set of questions for contributors to consider when evaluating an article following a framework often used for assessing medical literature.
+This approach emphasizes examining the methods used, assignment (whether the study was observational or randomized), assessment, results, interpretation, and how well the study extrapolates [@doi:10.5014/ajot.60.4.367].
+Contributors were also invited to contribute or edit text using GitHub's pull request system.
+These contributions were not strictly defined, and could range from minor corrections to punctuation and grammar to large-scale additions of text.
+Each pull request was reviewed and approved by at least one other contributor before being merged into the main branch.
+We sought to tag potential reviewers based on the introductions they had contributed in order to encourage participation.
+The emphasis on issues and pull requests was designed to encourage authors to discuss papers in the issues and to provide feedback (both formal and informal) on proposed text additions or changes.
+We also used gitter [@url:https://gitter.im] to promote informal questions and sharing of information among collaborators.
 
 #### Applying Manubot's Existing Capabilities to the Challenges of COVID-19 
 
@@ -113,6 +108,8 @@ The template variables also include versioned URLs to the dynamically updated fi
 The JSON files and figures are stored in the `external-resources` branch of the manuscript's GitHub repository, which acts as versioned storage.
 The GitHub Actions workflow automatically adds and commits the new JSON files and figures to the `external-resources` branch every time it runs, and Manubot uses the latest version of these resources when it builds the manuscript.
 
+Manubot also provides a functionality to create a bibliography using digital object identifiers (DOIs), website URLs, or other identifiers such as PubMed identifiers and arXiv IDs.
+
 #### Updating to Manubot in Response to Project Demands
 
 Due to the needs of this project, project contributors also implemented new features in Manubot.
@@ -136,14 +133,21 @@ Because in this implementation of Manubot, most collaborators were writing and e
 
 #### Recruitment and Manuscript Development
 
+We received a large 
+Because the GitHub issues and comment systems are relatively similar to other web common web activities, we found that authors were able to learn these tools fairly quickly.
+Similarly, the gitter chat also presented a low barrier to entry.
 <!--To Do: Add a graph/text describing number of unique contributors & commits over time-->
 <!-- To Do: Word Cloud of #17 to show diversity of interests?-->
 Appendix A contains summaries written by the students, post-docs, and faculty of the Immunology Institute at the Mount Sinai School of Medicine [@url:https://github.com/ismms-himc/covid-19_sinai_reviews; @doi:10.1038/s41577-020-0319-0], and two of the authors were recruited through the American Physician Scientist Association's Virtual Summer Research Program [@url:https://www.physicianscientists.org/page/summer-research-pilot-program].
 <!--To Do: Graph of contributions by individual over time and/or by career stage?-->
 <!-- To Do: unique contributors and commits over time like this: https://greenelab.github.io/meta-review/#fig:contrib-->
 <!--To Do: Other possible graphs: plot number of article issues opened over time, distribution of review contents, that is, something showing the number of comments per pull request that was merged, growth of the reference list over time and/or number of clinical trials cited (because we describe that as a necessary enhancement)-->
 
-<!--To Do: Links to documentation-->
+In order to make the project accessible to individuals from a number of backgrounds, we developed resources explaining how to use GitHub's web interface to develop and edit text and interact with Manubot for individuals with no prior experience working with git or other version control platforms.
+We developed tutorials containing visuals to explain how to open an issue, open a pull request, and review a pull request [@url:https://github.com/greenelab/covid19-review/blob/master/CONTRIBUTING.md; @url:https://github.com/greenelab/covid19-review/blob/master/INSTRUCTIONS.md].
+
+Articles were classified as _diagnostic_, _therapeutic_, or _other_, and a template was developed to guide the review of papers and preprints in each category.
+For examples of each template, please see Appendices B-D.
 
 #### Data Analysis and Integration
 
@@ -163,13 +167,15 @@ Using CI, Manubot now checks that the manuscript was built correctly, runs spell
 
 ### DISCUSSION
 
+The current project was managed through GitHub [@url:https://github.com/greenelab/covid19-review] using Manubot [@doi:10.1371/journal.pcbi.1007128] to continuously generate a version of the manuscript online [@url:https://greenelab.github.io/covid19-review].
 Working within the Manubot framework allowed for the successful facilitation of a massive collaborative review on an urgent topic.
 Developing Manubot for the specific challenges raised by COVID-19 and expanding on both training resources resulted in seven evolving literature reviews produced by the COVID-19 Review Consortium, all of which are currently available through Manubot and, in some cases, on arXiv <!--To Do: cite individual manuscripts-->.
 As many other efforts have described, the rate of publishing of formal manuscripts and preprints about COVID-19 has been unprecedented [@doi:10.1053/j.ackd.2020.08.003].
 The Manubot framework will allow for continuous updating of the manuscripts as the pandemic enters its second year and the landscape shifts with the emergence of promising therapeutics and vaccines [@individual-therapeutics; @individual-vaccines].
 The manuscripts pull data from XX data sources, allowing for information and visualizations to be updated daily using CI.
 This computational approach allows for the information in the manuscripts to be kept up to date automatically.
 
+Given the limitations imposed upon scientists by the COVID-19 pandemic and social distancing measures that had most scientists (including students) working from home for much of 2020, community building across disciplines and across career stages was a priority of the project.
 Beyond the immediate goal of applying Manubot to the challenges of COVID-19, we have also expanded Manubot to allow for broader participation in open publishing from fields where computational training in tools like version control is uncommon. 
 However, the broader topic of COVID-19 intersects with a wide range of fields, including virology, immunology, medicine, pharmacology, evolutionary biology, public health, and more, and any effort to comprehensively document and evaluate this body of literature would require insight from scientists across a number of fields.
 Furthermore, during the initial phase of the COVID-19 pandemic during spring and summer 2020, and much longer in some part of the world, many biological scientists were unable to access their research spaces.