Merge branch 'develop' into 2202-file-type-facet-fix #2202

IQSS · May 16, 2019 · f95a627 · f95a627
2 parents 4f91e9a + 0dc7c91
commit f95a627
Show file tree

Hide file tree

Showing 96 changed files with 171,288 additions and 551 deletions.
diff --git a/.gitignore b/.gitignore
@@ -2,6 +2,9 @@ nb-configuration.xml
 target
 infer-out
 nbactions.xml
+.settings
+.classpath
+.project
 michael-local
 GPATH
 GTAGS

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -57,7 +57,7 @@ If you are interested in working on the main Dataverse code, great! Before you s
 
 Please read http://guides.dataverse.org/en/latest/developers/version-control.html to understand how we use the "git flow" model of development and how we will encourage you to create a GitHub issue (if it doesn't exist already) to associate with your pull request. That page also includes tips on making a pull request.
 
-After making your pull request, your goal should be to help it advance through our kanban board at https://waffle.io/IQSS/dataverse . If no one has moved your pull request to the code review column in a timely manner, please reach out. Thanks!
+After making your pull request, your goal should be to help it advance through our kanban board at https://github.com/orgs/IQSS/projects/2 . If no one has moved your pull request to the code review column in a timely manner, please reach out. Thanks!
 
 [dataverse-community Google Group]: https://groups.google.com/group/dataverse-community
 [Community Call]: https://dataverse.org/community-calls

diff --git a/PULL_REQUEST_TEMPLATE.md b/PULL_REQUEST_TEMPLATE.md
@@ -4,7 +4,7 @@ Welcome! New contributors should at least glance at [CONTRIBUTING.md](/CONTRIBUT
 
 ## Related Issues
 
-- connects to #ISSUE_NUMBER: ISSUE_TITLE
+- #ISSUE_NUMBER: ISSUE_TITLE
 
 ## Pull Request Checklist
 

diff --git a/README.md b/README.md
@@ -1,7 +1,7 @@
 Dataverse&#174; 
 ===============
 
-Dataverse is an [open source][] web application for sharing, citing, analyzing, and preserving research data (developed by the [Data Science and Products team](http://www.iq.harvard.edu/people/people/data-science-products) at the [Institute for Quantitative Social Science](http://iq.harvard.edu/) and the [Dataverse community][]).
+Dataverse is an [open source][] software platform for sharing, finding, citing, and preserving research data (developed by the [Data Science and Products team](http://www.iq.harvard.edu/people/people/data-science-products) at the [Institute for Quantitative Social Science](http://iq.harvard.edu/) and the [Dataverse community][]).
 
 [dataverse.org][] is our home on the web and shows a map of Dataverse installations around the world, a list of [features][], [integrations][] that have been made possible through [REST APIs][], our development [roadmap][], and more.
 
@@ -17,8 +17,6 @@ Dataverse is a trademark of President and Fellows of Harvard College and is regi
 
 [![Dataverse Project logo](src/main/webapp/resources/images/dataverseproject_logo.jpg?raw=true "Dataverse Project")](http://dataverse.org)
 
-[![Waffle.io - Columns and their card count](https://badge.waffle.io/IQSS/dataverse.svg?columns=all)](https://waffle.io/IQSS/dataverse)
-
 [![Build Status](https://travis-ci.org/IQSS/dataverse.svg?branch=develop)](https://travis-ci.org/IQSS/dataverse) [![Coverage Status](https://coveralls.io/repos/IQSS/dataverse/badge.svg?branch=develop&service=github)](https://coveralls.io/github/IQSS/dataverse?branch=develop)
 
 [dataverse.org]: https://dataverse.org

diff --git a/doc/release-notes/3575-usernames.md b/doc/release-notes/3575-usernames.md
@@ -0,0 +1,4 @@
+* In an effort to prevent accidental duplicate accounts, user spoofing, or other username-based confusion, this release introduces a database constraint that no longer allows usernames that are exactly the same but use different capitalization, e.g. Bob11 vs. bob11. You may need to do some cleanup before upgrading to deal with existing usernames like this.
+* To check whether you have any usernames like this that need cleaning up, run the case insensitive duplicate queries from our [Useful Queries doc](https://docs.google.com/document/d/1-Y_iUduSxdDNeK1yiGUxe7t-Md7Fy965jp4o4m1XEoE/edit?usp=sharing "Useful Queries doc").
+* Once you identify the usernames that need cleaning up, you should use either [Merge User Accounts](http://guides.dataverse.org/en/latest/api/native-api.html#merge-user-accounts) (if it’s the same person) or [Change User Identifier](http://guides.dataverse.org/en/latest/api/native-api.html#change-user-identifier) (if they are different people).
+* After the cleanup you can safely upgrade without issue.
diff --git a/doc/sphinx-guides/source/admin/dashboard.rst b/doc/sphinx-guides/source/admin/dashboard.rst
@@ -29,3 +29,7 @@ Users
 
 This dashboard tool allows you to search a list of all users of your Dataverse installation. You can remove roles from user accounts and assign or remove superuser status. See the :doc:`user-administration` section for more details.
 
+Move Data
+---------
+
+This tool allows you to move datasets. To move dataverses, see the :doc:`dataverses-datasets` section.
diff --git a/doc/sphinx-guides/source/admin/dataverses-datasets.rst b/doc/sphinx-guides/source/admin/dataverses-datasets.rst
@@ -46,7 +46,9 @@ Datasets
 Move a Dataset
 ^^^^^^^^^^^^^^
 
-Moves a dataset whose id is passed to a dataverse whose alias is passed. If the moved dataset has a guestbook or a dataverse link that is not compatible with the destination dataverse, you will be informed and given the option to force the move and remove the guestbook or link. Only accessible to users with permission to publish the dataset in the original and destination dataverse. ::
+Superusers can move datasets using the dashboard. See also :doc:`dashboard`.
+
+Moves a dataset whose id is passed to a dataverse whose alias is passed. If the moved dataset has a guestbook or a dataverse link that is not compatible with the destination dataverse, you will be informed and given the option to force the move (with ``forceMove=true`` as a query parameter) and remove the guestbook or link (or both). Only accessible to users with permission to publish the dataset in the original and destination dataverse. ::
 
     curl -H "X-Dataverse-key: $API_TOKEN" -X POST http://$SERVER/api/datasets/$id/move/$alias
 
@@ -64,8 +66,23 @@ Removes a link between a dataset and a dataverse. Only accessible to superusers.
 
     curl -H "X-Dataverse-key: $API_TOKEN" -X DELETE http://$SERVER/api/datasets/$linked-dataset-id/deleteLink/$linking-dataverse-alias
 
-Mint new PID for a Dataset
-^^^^^^^^^^^^^^^^^^^^^^^^^^
+Mint a PID for a File That Does Not Have One
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In the following example, the database id of the file is 42::
+
+    export FILE_ID=42
+    curl http://localhost:8080/api/admin/$FILE_ID/registerDataFile
+
+Mint PIDs for Files That Do Not Have Them
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you have a large number of files, you might want to consider miniting PIDs for files individually using the ``registerDataFile`` endpoint above in a for loop, sleeping between each registration::
+
+    curl http://localhost:8080/api/admin/registerDataFileAll
+
+Mint a New DOI for a Dataset with a Handle
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Mints a new identifier for a dataset previously registered with a handle. Only accessible to superusers. ::
 

diff --git a/doc/sphinx-guides/source/admin/harvestserver.rst b/doc/sphinx-guides/source/admin/harvestserver.rst
@@ -14,6 +14,14 @@ harvesting protocol. Note that the terms "Harvesting Server" and "OAI
 Server" are being used interchangeably throughout this guide and in
 the inline help text.
 
+If you want to learn more about OAI-PMH, you could take a look at
+`DataCite OAI-PMH guide <https://support.datacite.org/docs/datacite-oai-pmh>`_
+or the `OAI-PMH protocol definition <https://www.openarchives.org/OAI/openarchivesprotocol.html>`_.
+
+You might consider adding your OAI-enabled production instance of Dataverse to
+`this shared list <https://docs.google.com/spreadsheets/d/12cxymvXCqP_kCsLKXQD32go79HBWZ1vU_tdG4kvP5S8/>`_
+of such instances.
+
 How does it work? 
 -----------------
 
@@ -28,6 +36,10 @@ Harvesting server can be enabled or disabled on the "Harvesting
 Server" page accessible via the :doc:`dashboard`. Harvesting server is by
 default disabled on a brand new, "out of the box" Dataverse.
 
+The OAI-PMH endpoint can be accessed at ``http(s)://<Your Dataverse FQDN>/oai``.
+If you want other services to harvest your repository, point them to this URL.
+*Example URL for 'Identify' verb*: `demo.dataverse.org OAI <https://demo.dataverse.org/oai?verb=Identify>`_
+
 OAI Sets
 --------
 

diff --git a/doc/sphinx-guides/source/admin/metadataexport.rst b/doc/sphinx-guides/source/admin/metadataexport.rst
@@ -7,14 +7,7 @@ Metadata Export
 Automatic Exports
 -----------------
 
-Publishing a dataset automatically starts a metadata export job, that will run in the background, asynchronously. Once completed, it will make the dataset metadata exported and cached in all the supported formats:
-
-- Dublin Core
-- Data Documentation Initiative (DDI)
-- DataCite 4
-- native JSON (Dataverse-specific)
-- OAI_ORE
-- Schema.org JSON-LD
+Publishing a dataset automatically starts a metadata export job, that will run in the background, asynchronously. Once completed, it will make the dataset metadata exported and cached in all the supported formats listed under :ref:`Supported Metadata Export Formats <metadata-export-formats>` in the :doc:`/user/dataset-management` section of the User Guide.
 
 A scheduled timer job that runs nightly will attempt to export any published datasets that for whatever reason haven't been exported yet. This timer is activated automatically on the deployment, or restart, of the application. So, again, no need to start or configure it manually. (See the "Application Timers" section of this guide for more information)
 

diff --git a/doc/sphinx-guides/source/api/native-api.rst b/doc/sphinx-guides/source/api/native-api.rst
@@ -291,7 +291,7 @@ Export Metadata of a Dataset in Various Formats
 
     GET http://$SERVER/api/datasets/export?exporter=ddi&persistentId=$persistentId
 
-.. note:: Supported exporters (export formats) are ``ddi``, ``oai_ddi``, ``dcterms``, ``oai_dc``, ``schema.org`` , ``OAI_ORE`` , ``Datacite`` and ``dataverse_json``.
+.. note:: Supported exporters (export formats) are ``ddi``, ``oai_ddi``, ``dcterms``, ``oai_dc``, ``schema.org`` , ``OAI_ORE`` , ``Datacite``, ``oai_datacite`` and ``dataverse_json``.
 
 Schema.org JSON-LD
 ^^^^^^^^^^^^^^^^^^
@@ -572,7 +572,7 @@ Optionally, you can check if there's a lock of a specific type on the dataset::
 
     curl "$SERVER_URL/api/datasets/{database_id}/locks?type={lock_type}
 
-Currently implemented lock types are ``Ingest, Workflow, InReview, DcmUpload and pidRegister``. 
+Currently implemented lock types are ``Ingest, Workflow, InReview, DcmUpload, pidRegister, and EditInProgress``. 
 
 The API will output the list of locks, for example:: 
 

diff --git a/doc/sphinx-guides/source/conf.py b/doc/sphinx-guides/source/conf.py
@@ -65,9 +65,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '4.13'
+version = '4.14'
 # The full version, including alpha/beta/rc tags.
-release = '4.13'
+release = '4.14'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

diff --git a/doc/sphinx-guides/source/developers/coding-style.rst b/doc/sphinx-guides/source/developers/coding-style.rst
@@ -89,11 +89,18 @@ Generally speaking you should use ``fine`` for everything that you don't want to
 
 When adding logging, do not simply add ``System.out.println()`` lines because the logging level cannot be controlled.
 
-Avoid Hard-Coding Strings
-~~~~~~~~~~~~~~~~~~~~~~~~~
+Avoid Hard-Coding Strings (Use Constants)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Special strings should be defined as public constants. For example, ``DatasetFieldConstant.java`` contains a field for "title" and it's used in many places in the code (try "Find Usages" in Netbeans). This is better than writing the string "title" in all those places.
 
+Avoid Hard-Coding User-Facing Messaging in English
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There is an ongoing effort to translate Dataverse into various languages. Look for "lang" or "languages" in the :doc:`/installation/config` section of the Installation Guide for details if you'd like to help or play around with this feature.
+
+The translation effort is hampered if you hard code user-facing messages in English in the Java code. Put English strings in ``Bundle.properties`` and use ``BundleUtil`` to pull them out. This is especially important for messages that appear in the UI. We are aware that the API has many, many hard coded English strings in it. If you touch a method in the API and notice English strings, you are strongly encouraged to used that opportunity to move the English to ``Bundle.properties``.
+
 Type Safety
 ~~~~~~~~~~~
 

diff --git a/doc/sphinx-guides/source/developers/intro.rst b/doc/sphinx-guides/source/developers/intro.rst
@@ -34,7 +34,7 @@ For the Dataverse development roadmap, please see https://dataverse.org/goals-ro
 Kanban Board
 ------------
 
-You can get a sense of what's currently in flight (in dev, in QA, etc.) by looking at https://waffle.io/IQSS/dataverse
+You can get a sense of what's currently in flight (in dev, in QA, etc.) by looking at https://github.com/orgs/IQSS/projects/2
 
 Issue Tracker
 -------------

diff --git a/doc/sphinx-guides/source/developers/version-control.rst b/doc/sphinx-guides/source/developers/version-control.rst
@@ -62,7 +62,7 @@ For guidance on which issue to work on, please ask! Also, see https://github.com
 
 Let's say you want to tackle https://github.com/IQSS/dataverse/issues/3728 which points out a typo in a page of Dataverse's documentation.
 
-If you tell us your GitHub username we are happy to add you to the "read only" team at https://github.com/orgs/IQSS/teams/dataverse-readonly/members so that we can assign the issue to you while you're working on it. You can also tell us if you'd like to be added to the `Dataverse Community Contributors spreadsheet <https://docs.google.com/spreadsheets/d/1o9DD-MQ0WkrYaEFTD5rF_NtyL8aUISgURsAXSL7Budk/edit?usp=sharing>`_ and the `Dev Efforts by the Dataverse Community spreadsheet <https://groups.google.com/d/msg/dataverse-community/X2diSWYll0w/ikp1TGcfBgAJ>`_.
+If you tell us your GitHub username we are happy to add you to the "read only" team at https://github.com/orgs/IQSS/teams/dataverse-readonly/members so that we can assign the issue to you while you're working on it. You can also tell us if you'd like to be added to the `Dataverse Community Contributors spreadsheet <https://docs.google.com/spreadsheets/d/1o9DD-MQ0WkrYaEFTD5rF_NtyL8aUISgURsAXSL7Budk/edit?usp=sharing>`_.
 
 Create a New Branch off the develop Branch
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -82,14 +82,14 @@ Push your feature branch to your fork of Dataverse. Your git command may look so
 Make a Pull Request
 ~~~~~~~~~~~~~~~~~~~
 
-Make a pull request to get approval to merge your changes into the develop branch. Feedback on the pull request template we use is welcome! The "connects to #3728" syntax is important because it's used at https://waffle.io/IQSS/dataverse to associate pull requests with issues.
+Make a pull request to get approval to merge your changes into the develop branch. Feedback on the pull request template we use is welcome!
 
 Here's an example of a pull request for issue #3728: https://github.com/IQSS/dataverse/pull/3827
 
 Make Sure Your Pull Request Has Been Advanced to Code Review
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Now that you've made your pull request, your goal is to make sure it appears in the "Code Review" column at https://waffle.io/IQSS/dataverse 
+Now that you've made your pull request, your goal is to make sure it appears in the "Code Review" column at https://github.com/orgs/IQSS/projects/2
 
 Look at https://github.com/IQSS/dataverse/blob/master/CONTRIBUTING.md for various ways to reach out to developers who have enough access to the GitHub repo to move your issue and pull request to the "Code Review" column.
 

diff --git a/doc/sphinx-guides/source/installation/config.rst b/doc/sphinx-guides/source/installation/config.rst
@@ -540,6 +540,84 @@ Once you have the location of your custom CSS file, run this curl command to add
 
 ``curl -X PUT -d '/var/www/dataverse/branding/custom-stylesheet.css' http://localhost:8080/api/admin/settings/:StyleCustomizationFile``
 
+.. _i18n:
+
+Internationalization
+--------------------
+
+Dataverse is being translated into multiple languages by the Dataverse community! Please see below for how to help with this effort!
+
+Adding Multiple Languages to the Dropdown in the Header
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+The presence of the :ref:`:Languages` database setting adds a dropdown in the header for multiple languages. For example to add English and French to the dropdown:
+
+``curl http://localhost:8080/api/admin/settings/:Languages -X PUT -d '[{"locale":"en","title":"English"},{"locale":"fr","title":"Français"}]'``
+
+Configuring the "lang" Directory
+++++++++++++++++++++++++++++++++
+
+Translations for Dataverse are stored in "properties" files in a directory on disk (e.g. ``/home/glassfish/langBundles``) that you specify with the :ref:`dataverse.lang.directory` ``dataverse.lang.directory`` JVM option, like this:
+
+``./asadmin create-jvm-options '-Ddataverse.lang.directory=/home/glassfish/langBundles'``
+
+Go ahead and create the directory you specified.
+
+``mkdir /home/glassfish/langBundles``
+
+Creating a languages.zip File
++++++++++++++++++++++++++++++
+
+Dataverse provides and API endpoint for adding languages using a zip file.
+
+First, clone the "dataverse-language-packs" git repo.
+
+``git clone https://github.com/GlobalDataverseCommunityConsortium/dataverse-language-packs.git``
+
+Take a look at https://github.com/GlobalDataverseCommunityConsortium/dataverse-language-packs/branches to see if the version of Dataverse you're running has translations.
+
+Change to the directory for the git repo you just cloned.
+
+``cd dataverse-language-packs``
+
+Switch (``git checkout``) to the branch based on Dataverse version you are running. The branch "dataverse-v4.13" is used in the example below.
+
+``export BRANCH_NAME=dataverse-v4.13``
+
+``git checkout $BRANCH_NAME``
+
+Create a "languages" directory in "/tmp".
+
+``mkdir /tmp/languages``
+
+Copy the properties files into the "languages" directory
+
+``cp -R en_US/*.properties /tmp/languages``
+
+``cp -R fr_CA/*.properties /tmp/languages``
+
+Create the zip file
+
+``cd /tmp/languages``
+
+``zip languages.zip *.properties``
+
+Load the languages.zip file into Dataverse
+++++++++++++++++++++++++++++++++++++++++++
+
+Now that you have a "languages.zip" file, you can load it into Dataverse with the command below.
+
+``curl http://localhost:8080/api/admin/datasetfield/loadpropertyfiles -X POST --upload-file /tmp/languages/languages.zip -H "Content-Type: application/zip"``
+
+Click on the languages using the drop down in the header to try them out.
+
+How to Help Translate Dataverse Into Your Language
+++++++++++++++++++++++++++++++++++++++++++++++++++
+
+Please join the `dataverse-internationalization-wg`_ mailing list and contribute to https://github.com/GlobalDataverseCommunityConsortium/dataverse-language-packs to help translate Dataverse into various languages!
+
+.. _dataverse-internationalization-wg: https://groups.google.com/forum/#!forum/dataverse-internationalization-wg
+
 .. _Web-Analytics-Code:
 
 Web Analytics Code
@@ -897,13 +975,13 @@ This JVM option is only relevant if you plan to run multiple Glassfish servers f
 dataverse.lang.directory
 ++++++++++++++++++++++++
 
-This JVM option is used to configure the path where all the language specific property files are to be stored.  If this option is set then the english property file must be present in the path along with any other language property file. You can download language property files from https://github.com/GlobalDataverseCommunityConsortium/dataverse-language-packs
+This JVM option is used to configure the path where all the language specific property files are to be stored.  If this option is set then the English property file must be present in the path along with any other language property file. You can download language property files from https://github.com/GlobalDataverseCommunityConsortium/dataverse-language-packs
 
 ``./asadmin create-jvm-options '-Ddataverse.lang.directory=PATH_LOCATION_HERE'``
 
 If this value is not set, by default, a Dataverse installation will read the English language property files from the Java Application.
 
-See also the ``:Languages`` setting below.
+See also :ref:`i18n`.
 
 dataverse.files.hide-schema-dot-org-download-urls
 +++++++++++++++++++++++++++++++++++++++++++++++++
@@ -1633,15 +1711,15 @@ Sets the path where the raw Make Data Count logs are stored before being process
 
 ``curl -X PUT -d '/usr/local/glassfish4/glassfish/domains/domain1/logs' http://localhost:8080/api/admin/settings/:MDCLogPath``
 
+.. _:Languages:
+
 :Languages
 ++++++++++
 
 Sets which languages should be available. If there is more than one, a dropdown is displayed
-in the header. This should be formated as a JSON array as shown below.
-
-``curl http://localhost:8080/api/admin/settings/:Languages -X PUT -d '[{  "locale":"en", "title":"English"},  {  "locale":"fr", "title":"Français"}]'``
+in the header.
 
-See also the ``dataverse.lang.directory`` JVM option above.
+See :ref:`i18n` for a curl example and related settings.
 
 :InheritParentRoleAssignments
 +++++++++++++++++++++++++++++