Merge branch 'develop' into 5867-api-for-hash-verification-regeneration

CONTRIBUTING.md

@@ -8,7 +8,7 @@ We aren't just looking for developers. There are many ways to contribute to Data
 
 Your idea or feature request might already be captured in the Dataverse [issue tracker] on GitHub but if not, the best way to bring it to the community's attention is by posting on the [dataverse-community Google Group][] or bringing it up on a [Community Call][]. You're also welcome make some noise in the [#dataverse IRC channel][] (which is [logged][]) or cram your idea into 280 characters and mention [@dataverseorg][] on Twitter. To discuss your idea privately, please email it to support@dataverse.org
 
-There's a chance your idea is already on our roadmap, which is available at http://dataverse.org/goals-roadmap-and-releases
+There's a chance your idea is already on our roadmap, which is available at https://www.iq.harvard.edu/roadmap-dataverse-project
 
 [#dataverse IRC channel]: http://chat.dataverse.org
 [logged]: http://irclog.iq.harvard.edu/dataverse/today
@@ -29,6 +29,8 @@ If there is no pre-existing issue or it has been closed, please click on the "Ne
 
 If you do not receive a reply to your new issue or comment in a timely manner, please email support@dataverse.org with a link to the issue.
 
+We are aware of the new issue templates described at https://help.github.com/articles/about-issue-and-pull-request-templates but haven't converted over yet.
+
 ### Writing an Issue
 
 For the subject of an issue, please start it by writing the feature or functionality it relates to, i.e. "Create Account:..." or "Dataset Page:...". In the body of the issue, please outline the issue you are reporting with as much detail as possible. In order for the Dataverse development team to best respond to the issue, we need as much information about the issue as you can provide. Include steps to reproduce bugs. Indicate which version you're using, which is shown at the bottom of the page. We love screenshots!
@@ -48,7 +50,7 @@ The source for the documentation at http://guides.dataverse.org/en/latest/ is in
 
 ## Code/Pull Requests
 
-We love code contributions. Developers are not limited to the main Dataverse code in this git repo. You can help with API client libraries in your favorite language that are mentioned in the [API Guide][] or create a new library. You can help work on configuration management code that's mentioned in the [Installation Guide][]. The Installation Guide also covers a new concept called "external tools" that allows developers to create their own tools that are available from within an installation of Dataverse.
+We love code contributions. Developers are not limited to the main Dataverse code in this git repo. You can help with API client libraries in your favorite language that are mentioned in the [API Guide][] or create a new library. You can help work on configuration management code that's mentioned in the [Installation Guide][]. The Installation Guide also covers a relatively new concept called "external tools" that allows developers to create their own tools that are available from within an installation of Dataverse.
 
 [API Guide]: http://guides.dataverse.org/en/latest/api
 [Installation Guide]: http://guides.dataverse.org/en/latest/installation
@@ -57,11 +59,13 @@ If you are interested in working on the main Dataverse code, great! Before you s
 
 Please read http://guides.dataverse.org/en/latest/developers/version-control.html to understand how we use the "git flow" model of development and how we will encourage you to create a GitHub issue (if it doesn't exist already) to associate with your pull request. That page also includes tips on making a pull request.
 
-After making your pull request, your goal should be to help it advance through our kanban board at https://github.com/orgs/IQSS/projects/2 . If no one has moved your pull request to the code review column in a timely manner, please reach out. Thanks!
+After making your pull request, your goal should be to help it advance through our kanban board at https://github.com/orgs/IQSS/projects/2 . If no one has moved your pull request to the code review column in a timely manner, please reach out. Note that once a pull request is created for an issue, we'll remove the issue from the board so that we only track one card (the pull request).
+
+Thanks for your contribution!
 
 [dataverse-community Google Group]: https://groups.google.com/group/dataverse-community
 [Community Call]: https://dataverse.org/community-calls
 [dataverse-dev Google Group]: https://groups.google.com/group/dataverse-dev
 [IRC]: http://chat.dataverse.org
 [community contributors]: https://docs.google.com/spreadsheets/d/1o9DD-MQ0WkrYaEFTD5rF_NtyL8aUISgURsAXSL7Budk/edit?usp=sharing
-[dev efforts]: https://groups.google.com/d/msg/dataverse-community/X2diSWYll0w/ikp1TGcfBgAJ
+[dev efforts]: https://github.com/orgs/IQSS/projects/2#column-5298405

ISSUE_TEMPLATE.md

@@ -1 +1 @@
-Thank you for contributing an issue to the Dataverse Project! If this is a bug report, please let us know when the issue occurs, which page it occurs on, to whom it occurs, and which version of Dataverse you're using. If this is a feature request, please let us know what you'd like to see and give us some context - what kind of user is the feature intended for, and what inspired the request? No matter the issue, screenshots are always welcome.
+Thank you for contributing an issue to the Dataverse Project! If this is a bug report, please let us know when the issue occurs, which page it occurs on, to whom it occurs, and which version of Dataverse you're using. If this is a feature request, please let us know what you'd like to see and give us some context - what kind of user is the feature intended for, the relevant use cases, and what inspired the request? No matter the issue, screenshots are always welcome.

PULL_REQUEST_TEMPLATE.md

@@ -1,10 +1,12 @@
 ## New Contributors
 
-Welcome! New contributors should at least glance at [CONTRIBUTING.md](/CONTRIBUTING.md), especially the section on pull requests where we encourage you to reach out to other developers before you start coding. Also, please note that we measure code coverage and prefer you write unit tests. Pull requests can still be reviewed without tests or completion of the checklist outlined below. Thanks!
+Welcome! New contributors should at least glance at [CONTRIBUTING.md](/CONTRIBUTING.md), especially the section on pull requests where we encourage you to reach out to other developers before you start coding. Also, please note that we measure code coverage and prefer you write unit tests. Pull requests can still be reviewed without tests or completion of the checklist outlined below. Note that we use the "closes" syntax below to trigger Github's automation to close the corresponding issue once the pull request is merged.
+
+Thanks for your contribution to Dataverse!
 
 ## Related Issues
 
-- #ISSUE_NUMBER: ISSUE_TITLE
+- closes #ISSUE_NUMBER: ISSUE_TITLE
 
 ## Pull Request Checklist
 

README.md

@@ -25,7 +25,7 @@ Dataverse is a trademark of President and Fellows of Harvard College and is regi
 [Installation Guide]: http://guides.dataverse.org/en/latest/installation/index.html
 [latest release]: https://github.com/IQSS/dataverse/releases
 [features]: https://dataverse.org/software-features
-[roadmap]: https://dataverse.org/goals-roadmap-and-releases
+[roadmap]: https://www.iq.harvard.edu/roadmap-dataverse-project
 [integrations]: https://dataverse.org/integrations
 [REST APIs]: http://guides.dataverse.org/en/latest/api/index.html
 [Contributing Guide]: CONTRIBUTING.md

doc/sphinx-guides/requirements.txt

@@ -1,2 +1,2 @@
 # match version used by Jenkins
-Sphinx==1.1.3
+Sphinx==1.5.6

doc/sphinx-guides/source/admin/integrations.rst

@@ -112,8 +112,10 @@ For details on how to configure this integration, look for "DuraCloud/Chronopoli
 Future Integrations
 -------------------
 
-The `Dataverse roadmap <https://dataverse.org/goals-roadmap-and-releases>`_ is a good place to see integrations that the core Dataverse team is working on.
+The `Dataverse roadmap <https://www.iq.harvard.edu/roadmap-dataverse-project>`_ is a good place to see integrations that the core Dataverse team is working on.
 
-The `Dev Efforts by the Dataverse Community <https://docs.google.com/spreadsheets/d/1pl9U0_CtWQ3oz6ZllvSHeyB0EG1M_vZEC_aZ7hREnhE/edit?usp=sharing>`_ spreadsheet is the best way to track integrations that are being worked on by the Dataverse community.
+The `Community Dev <https://github.com/orgs/IQSS/projects/2#column-5298405>`_ column of our project board is a good way to track integrations that are being worked on by the Dataverse community but many are not listed and if you have an idea for an integration, please ask on the `dataverse-community <https://groups.google.com/forum/#!forum/dataverse-community>`_ mailing list if someone is already working on it.
+
+Many integrations take the form of "external tools". See the :doc:`external-tools` section for details. External tool makers should check out the :doc:`/api/external-tools` section of the API Guide.
 
 Please help us keep this page up to date making a pull request! To get started, see the :doc:`/developers/documentation` section of the Developer Guide.

doc/sphinx-guides/source/developers/coding-style.rst

@@ -99,7 +99,7 @@ Avoid Hard-Coding User-Facing Messaging in English
 
 There is an ongoing effort to translate Dataverse into various languages. Look for "lang" or "languages" in the :doc:`/installation/config` section of the Installation Guide for details if you'd like to help or play around with this feature.
 
-The translation effort is hampered if you hard code user-facing messages in English in the Java code. Put English strings in ``Bundle.properties`` and use ``BundleUtil`` to pull them out. This is especially important for messages that appear in the UI. We are aware that the API has many, many hard coded English strings in it. If you touch a method in the API and notice English strings, you are strongly encouraged to used that opportunity to move the English to ``Bundle.properties``.
+The translation effort is hampered if you hard code user-facing messages in English in the Java code. Put English strings in ``Bundle.properties`` and use ``BundleUtil`` to pull them out. This is especially important for messages that appear in the UI. We are aware that the API has many, many hard coded English strings in it. If you touch a method in the API and notice English strings, you are strongly encouraged to use that opportunity to move the English to ``Bundle.properties``.
 
 Type Safety
 ~~~~~~~~~~~

doc/sphinx-guides/source/developers/intro.rst

@@ -31,7 +31,7 @@ We make use of a variety of Java EE technologies such as JPA, JAX-RS, JMS, and J
 Roadmap
 -------
 
-For the Dataverse development roadmap, please see https://dataverse.org/goals-roadmap-and-releases
+For the Dataverse development roadmap, please see https://www.iq.harvard.edu/roadmap-dataverse-project
 
 Kanban Board
 ------------

doc/sphinx-guides/source/developers/making-releases.rst

@@ -46,12 +46,19 @@ The "develop" branch should be merged into "master" before tagging. See also the
 Write Release Notes
 -------------------
 
-Create a draft release at https://github.com/IQSS/dataverse/releases/new
+Developers should express the need for an addition to release notes by creating a file in ``/doc/release-notes`` containing the name of the issue they're working on. The name of the branch could be used for the filename with ".md" appended (release notes are written in Markdown) such as ``5053-apis-custom-homepage.md``. 
 
-- The "tag version" and "title" should be the number of the milestone with a "v" in front (i.e. v4.6.2).
-- For the description, follow previous examples at https://github.com/IQSS/dataverse/releases
+At or near release time:
+
+- Create an issue in Github to track the work of creating release notes for the upcoming release
+- Create a branch, add a .md file for the release (ex. 4.16 Release Notes) in ``/doc/release-notes`` and write the release notes, making sure to pull content from the issue-specific release notes mentioned above
+- Delete the previously-created, issue-specific release notes as the content is added to the main release notes file
+- Take the release notes .md through the regular Code Review and QA process
+- Create a draft release at https://github.com/IQSS/dataverse/releases/new
+- The "tag version" and "title" should be the number of the milestone with a "v" in front (i.e. v4.16).
+- Copy in the content from the .md file
+- For the description, follow post-4.16 examples at https://github.com/IQSS/dataverse/releases
 
-Please note that the current process involves copying and pasting a running Google doc into release notes but we are conducting an experiment whereby developers can express the need for an addition to release notes by creating a file in ``/doc/release-notes`` containing the name of the issue they're working on. Perhaps the name of the branch could be used for the filename with ".md" appended (release notes are written in Markdown) such as ``5053-apis-custom-homepage.md``. To avoid accumulating many stale files over time, when a release is cut these files should probably be removed with ``git rm``.
 
 Make Artifacts Available for Download
 -------------------------------------
@@ -60,7 +67,7 @@ Upload the following artifacts to the draft release you created:
 
 - war file (``mvn package`` from Jenkins)
 - installer (``cd scripts/installer && make``)
-- other files as needed, such as an updated Solr schema
+- other files as needed, such as updated Solr schema and config files
 
 Publish Release
 ---------------

doc/sphinx-guides/source/developers/version-control.rst

@@ -82,14 +82,14 @@ Push your feature branch to your fork of Dataverse. Your git command may look so
 Make a Pull Request
 ~~~~~~~~~~~~~~~~~~~
 
-Make a pull request to get approval to merge your changes into the develop branch. Feedback on the pull request template we use is welcome!
+Make a pull request to get approval to merge your changes into the develop branch. Note that once a pull request is created, we'll remove the corresponding issue from our kanban board so that we're only tracking one card.
 
-Here's an example of a pull request for issue #3728: https://github.com/IQSS/dataverse/pull/3827
+Feedback on the pull request template we use is welcome! Here's an example of a pull request for issue #3827: https://github.com/IQSS/dataverse/pull/3827
 
 Make Sure Your Pull Request Has Been Advanced to Code Review
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Now that you've made your pull request, your goal is to make sure it appears in the "Code Review" column at https://github.com/orgs/IQSS/projects/2
+Now that you've made your pull request, your goal is to make sure it appears in the "Code Review" column at https://github.com/orgs/IQSS/projects/2. 
 
 Look at https://github.com/IQSS/dataverse/blob/master/CONTRIBUTING.md for various ways to reach out to developers who have enough access to the GitHub repo to move your issue and pull request to the "Code Review" column.
 

scripts/tests/ec2-memory-benchmark/README.md

@@ -0,0 +1,59 @@
+## Automated Memory Benchmark Test
+
+In order to run the benchmark test you will need the following: 
+
+* The two scripts from this directory: `ec2-memory-benchmark-remote.sh` and `ec2-memory-benchmark.sh`
+
+* Have the aws command line utilities installed.
+
+* Have your AWS credentials configured, so that you can spin up EC2 instances. 
+
+All the other pieces will be downloaded automatically (from the
+`IQSS/dataverse-ansible` and `IQSS/dataverse-sample-data` GitHub repos),
+either locally, where you're running the script, or on the newly
+created instance.
+
+Run the script as follows: 
+
+```
+./ec2-memory-benchmark.sh [DATAVERSE_BRANCH]
+```
+
+Once the instance is created and populated with the data from
+`dataverse-sample-data`, the script will run a memory-intensive stress
+test and record some information about the memory utilisation. We are
+using the approach tested in #6035 (bombarding the system with GETs on
+select pages, then taking snapshots of the allocated classes on the
+heap and the garbage collection stats; see the issue for more info).
+
+In the curret implementation we are testing the two most used pages -
+dataverse and dataset. Once (and if) it completes, the script will
+provide links to the produced output (downloadable from the EC2
+instance).
+
+This should be considered work in progress. We'll be working on
+creating more tests and figuring out how much/what else to include in
+the reports.
+
+The goal, for now is to be able to test a release (or a branch) and
+produce a quantifiable benchmark that can be compared to the result
+from the previous release/parent branch.
+
+In addition to the raw text output the script will produce simple heap
+utilisation plots like the examples below:
+
+<div style="text-align:center; margin:2em">
+  <img src="benchmark-dataverse.png">
+</div>
+
+<div style="text-align:center; margin:2em">
+  <img src="benchmark-dataset.png">
+</div>
+
+The immediate value of the examples above is that they confirm
+visually that the memory allocated by the pages is recoverable by the
+garbage collector (the line drawn through the lowest points of each
+"saw tooth" is pointing down, not up). It will be even more useful to
+superimpose a graph like this on something produced by another
+release, for comparison.
+