Merge branch 'develop' into 3036-support-form-math-bug

.gitignore

@@ -22,4 +22,7 @@ scripts/api/py_api_wrapper/local-data/*
 doc/sphinx-guides/build
 faces-config.NavData
 src/main/java/BuildNumber.properties
-/nbproject/
+/nbproject/
+oauth-credentials.md
+
+/src/main/webapp/oauth2/newAccount.html

doc/Architecture/update-user-account-info.puml

@@ -0,0 +1,28 @@
+@startuml
+actor user
+participant DataverseUserPage as page
+participant "provider: AuthenticationProvider" as prv
+participant AuthenticationServiceBean as authSvc
+user --> page: HTTP GET
+== page init ==
+...
+page -> authSvc : lookupProvider(session.user)
+activate authSvc
+page <-- authSvc: provider
+deactivate authSvc
+...
+
+== Can Update User? ==
+page -> prv: isUserInfoUpdateAllowed()
+activate prv
+page <-- prv: true/false
+deactivate prv
+...
+
+== Save User ==
+user --> page: save()
+page -> page: udi = createUserDisplayInfo()
+page -> prv: updateUserInfo( udi )
+page -> authSvc: updateAuthentictedUser( session.user, udi )
+
+@enduml

doc/sphinx-guides/source/_static/installation/files/etc/shibboleth/shibAuthProvider.json

@@ -0,0 +1,5 @@
+{
+    "id":"shib",
+    "factoryAlias":"shib",
+    "enabled":true
+}

doc/sphinx-guides/source/_static/installation/files/root/auth-providers/github.json

@@ -0,0 +1,8 @@
+{
+    "id":"github",
+    "factoryAlias":"oauth2",
+    "title":"GitHub",
+    "subtitle":"",
+    "factoryData":"type: github | userEndpoint: NONE | clientId: FIXME | clientSecret: FIXME",
+    "enabled":true
+}

doc/sphinx-guides/source/_static/installation/files/root/auth-providers/google.json

@@ -0,0 +1,8 @@
+{
+    "id":"google",
+    "factoryAlias":"oauth2",
+    "title":"Google",
+    "subtitle":"",
+    "factoryData":"type: google | userEndpoint: NONE | clientId: FIXME | clientSecret: FIXME",
+    "enabled":true
+}

doc/sphinx-guides/source/_static/installation/files/root/auth-providers/orcid.json

@@ -0,0 +1,8 @@
+{
+    "id":"orcid",
+    "factoryAlias":"oauth2",
+    "title":"ORCID",
+    "subtitle":"",
+    "factoryData":"type: orcid | userEndpoint: https://api.orcid.org/v1.2/{ORCID}/orcid-profile | clientId: FIXME | clientSecret: FIXME",
+    "enabled":true
+}

doc/sphinx-guides/source/api/apps.rst

@@ -1,7 +1,7 @@
 Apps
 ====
 
-The introduction of Dataverse APIs has fostered the development of apps that are listed at http://datascience.iq.harvard.edu/collaborations
+The introduction of Dataverse APIs has fostered the development of apps that are listed at http://dataverse.org/integrations
 
 The apps below are open source, demonstrating how to use Dataverse APIs. Some of these apps (and others) are built on :doc:`/api/client-libraries` that are available for Dataverse APIs.
 
@@ -40,7 +40,7 @@ https://github.com/CenterForOpenScience/osf.io/tree/master/website/addons/datave
 GeoConnect
 ~~~~~~~~~~
 
-GeoConnect allows Dataverse files to be visualized on http://worldmap.harvard.edu with a "Map It" button.
+GeoConnect allows Dataverse files to be visualized on http://worldmap.harvard.edu with the "Explore" button. Read more about it in the :doc:`/user/data-exploration/worldmap` section of the User Guide.
 
 https://github.com/IQSS/geoconnect
 

doc/sphinx-guides/source/api/index.rst

@@ -13,6 +13,8 @@ APIs. In 4.0, we require to get a token, by simply registering for a Dataverse a
 
 Rather than using a production installation of Dataverse, API users are welcome to use http://demo.dataverse.org for testing.
 
+Please note that the APIs in this guide are shipped with the Dataverse software itself but additional APIs are available if you install the "miniverse" application from https://github.com/IQSS/miniverse and give it read only access to your production Dataverse database. http://dataverse.org/metrics is powered by miniverse.
+
 Contents:
 
 .. toctree::

doc/sphinx-guides/source/api/native-api.rst

@@ -3,6 +3,10 @@ Native API
 
 Dataverse 4.0 exposes most of its GUI functionality via a REST-based API. Some API calls do not require authentication. Calls that do require authentication require the user's API key. That key can be passed either via an extra query parameter, ``key``, as in ``ENPOINT?key=API_KEY``, or via the HTTP header ``X-Dataverse-key``. Note that while the header option normally requires more work on client side, it is considered safer, as the API key is not logged in the server access logs.
 
+.. note:: |CORS| Some API endpoint allow CORS_ (cross-origin resource sharing), which makes them usable from scripts runing in web browsers. These endpoints are marked with a *CORS* badge. 
+
+.. _CORS: https://www.w3.org/TR/cors/
+
 .. warning:: Dataverse 4.0's API is versioned at the URI - all API calls may include the version number like so: ``http://server-address//api/v1/...``. Omitting the ``v1`` part would default to the latest API version (currently 1). When writing scripts/applications that will be used for a long time, make sure to specify the API version, so they don't break when the API is upgraded.
 
 .. contents::
@@ -17,7 +21,7 @@ If ``$id`` is omitted, a root dataverse is created. ``$id`` can either be a data
 
     POST http://$SERVER/api/dataverses/$id?key=$apiKey
 
-The following JSON example can be `downloaded <../_static/api/dataverse-complete.json>`_ and modified to create dataverses to suit your needs. The fields ``name``, ``alias``, and ``dataverseContacts`` are required. The controlled vocabulary for ``dataverseType`` is
+Download the :download:`JSON example <../_static/api/dataverse-complete.json>` file and modified to create dataverses to suit your needs. The fields ``name``, ``alias``, and ``dataverseContacts`` are required. The controlled vocabulary for ``dataverseType`` is
 
 - ``JOURNALS``
 - ``LABORATORY``
@@ -30,23 +34,23 @@ The following JSON example can be `downloaded <../_static/api/dataverse-complete
 
 .. literalinclude:: ../_static/api/dataverse-complete.json
 
-View data about the dataverse identified by ``$id``. ``$id`` can be the id number of the dataverse, its alias, or the special value ``:root``. ::
+|CORS| View data about the dataverse identified by ``$id``. ``$id`` can be the id number of the dataverse, its alias, or the special value ``:root``. ::
 
     GET http://$SERVER/api/dataverses/$id
 
 Deletes the dataverse whose ID is given::
 
     DELETE http://$SERVER/api/dataverses/$id?key=$apiKey
 
-Lists all the DvObjects under dataverse ``id``. ::
+|CORS| Lists all the DvObjects under dataverse ``id``. ::
 
     GET http://$SERVER/api/dataverses/$id/contents
 
 All the roles defined directly in the dataverse identified by ``id``::
 
   GET http://$SERVER/api/dataverses/$id/roles?key=$apiKey
 
-List all the facets for a given dataverse ``id``. ::
+|CORS| List all the facets for a given dataverse ``id``. ::
 
   GET http://$SERVER/api/dataverses/$id/facets?key=$apiKey
 
@@ -73,7 +77,7 @@ Delete the assignment whose id is ``$id``::
 
   DELETE http://$SERVER/api/dataverses/$id/assignments/$id?key=$apiKey
 
-Get the metadata blocks defined on the passed dataverse::
+|CORS| Get the metadata blocks defined on the passed dataverse::
 
   GET http://$SERVER/api/dataverses/$id/metadatablocks?key=$apiKey
 
@@ -105,7 +109,7 @@ Publish the Dataverse pointed by ``identifier``, which can either by the dataver
 Datasets
 ~~~~~~~~
 
-**Note** Creation of new datasets is done with a ``POST`` onto dataverses. See dataverse section.
+**Note** Creation of new datasets is done with a ``POST`` onto dataverses. See Dataverses_ section.
 
 **Note** In all commands below, dataset versions can be referred to as:
 
@@ -126,38 +130,37 @@ Datasets
 
     GET http://$SERVER/api/datasets/:persistentId/versions/:draft?persistentId=doi:10.5072/FK2/J8SJZB
 
-Show the dataset whose id is passed::
+|CORS| Show the dataset whose id is passed::
 
   GET http://$SERVER/api/datasets/$id?key=$apiKey
 
 Delete the dataset whose id is passed::
 
   DELETE http://$SERVER/api/datasets/$id?key=$apiKey
 
-List versions of the dataset::
+|CORS| List versions of the dataset::
 
   GET http://$SERVER/api/datasets/$id/versions?key=$apiKey
 
-Show a version of the dataset. The Dataset also include any metadata blocks the data might have::
+|CORS| Show a version of the dataset. The Dataset also include any metadata blocks the data might have::
 
   GET http://$SERVER/api/datasets/$id/versions/$versionNumber?key=$apiKey
 
-
-Export the metadata of the current published version of a dataset in various formats see Note below::
+|CORS| Export the metadata of the current published version of a dataset in various formats see Note below::
 
     GET http://$SERVER/api/datasets/export?exporter=ddi&persistentId=$persistentId
 
 .. note:: Supported exporters (export formats) are ``ddi``, ``oai_ddi``, ``dcterms``, ``oai_dc``, and ``dataverse_json``.
 
-Lists all the file metadata, for the given dataset and version::
+|CORS| Lists all the file metadata, for the given dataset and version::
 
   GET http://$SERVER/api/datasets/$id/versions/$versionId/files?key=$apiKey
 
-Lists all the metadata blocks and their content, for the given dataset and version::
+|CORS| Lists all the metadata blocks and their content, for the given dataset and version::
 
   GET http://$SERVER/api/datasets/$id/versions/$versionId/metadata?key=$apiKey
 
-Lists the metadata block block named `blockname`, for the given dataset and version::
+|CORS| Lists the metadata block block named `blockname`, for the given dataset and version::
 
   GET http://$SERVER/api/datasets/$id/versions/$versionId/metadata/$blockname?key=$apiKey
 
@@ -204,7 +207,8 @@ Delete a Private URL from a dataset (if it exists)::
 Builtin Users
 ~~~~~~~~~~~~~
 
-This endopint deals with users of the built-in authentication provider. Note that users may come from other authentication services as well, such as Shibboleth.
+This endpoint deals with users of the built-in authentication provider. For more background on various authentication providers, see :doc:`/user/account` and :doc:`/installation/config`.
+
 For this service to work, the setting ``BuiltinUsers.KEY`` has to be set, and its value passed as ``key`` to
 each of the calls.
 
@@ -285,7 +289,7 @@ Management of Shibboleth groups via API is documented in the :doc:`/installation
 Info
 ~~~~
 
-Get the Dataverse version. The response contains the version and build numbers::
+|CORS| Get the Dataverse version. The response contains the version and build numbers::
 
   GET http://$SERVER/api/info/version
 
@@ -297,14 +301,15 @@ For now, only the value for the ``:DatasetPublishPopupCustomText`` setting from
 
   GET http://$SERVER/api/info/settings/:DatasetPublishPopupCustomText
 
+
 Metadata Blocks
 ~~~~~~~~~~~~~~~
 
-Lists brief info about all metadata blocks registered in the system::
+|CORS| Lists brief info about all metadata blocks registered in the system::
 
   GET http://$SERVER/api/metadatablocks
 
-Return data about the block whose ``identifier`` is passed. ``identifier`` can either be the block's id, or its name::
+|CORS| Return data about the block whose ``identifier`` is passed. ``identifier`` can either be the block's id, or its name::
 
   GET http://$SERVER/api/metadatablocks/$identifier
 
@@ -347,7 +352,13 @@ Show data about an authentication provider::
 
 Enable or disable an authentication provider (denoted by ``id``)::
 
-  POST http://$SERVER/api/admin/authenticationProviders/$id/:enabled
+  PUT http://$SERVER/api/admin/authenticationProviders/$id/enabled
+
+.. note:: The former endpoint, ending with ``:enabled`` (that is, with a colon), is still supported, but deprecated.
+
+Check whether an authentication proider is enabled::
+
+  GET http://$SERVER/api/admin/authenticationProviders/$id/enabled
 
 The body of the request should be either ``true`` or ``false``. Content type has to be ``application/json``, like so::
 
@@ -389,6 +400,21 @@ Sample output using "dataverseAdmin" as the ``identifier``::
       "affiliation": "Dataverse.org"
     }
 
+Create an authenticateUser::
+
+    POST http://$SERVER/api/admin/authenticatedUsers
+
+POSTed JSON example::
+
+    {
+      "authenticationProviderId": "orcid",
+      "persistentUserId": "0000-0002-3283-0661",
+      "identifier": "@pete",
+      "firstName": "Pete K.",
+      "lastName": "Dataversky",
+      "email": "pete@mailinator.com"
+    }
+
 Toggles superuser mode on the ``AuthenticatedUser`` whose ``identifier`` (without the ``@`` sign) is passed. ::
 
     POST http://$SERVER/api/admin/superuser/$identifier
@@ -455,3 +481,10 @@ Execute a saved search by database id and make links to dataverses and datasets
 Execute all saved searches and make links to dataverses and datasets that are found. ``debug`` works as described above.  ::
 
   PUT http://$SERVER/api/admin/savedsearches/makelinks/all?debug=true
+
+
+.. |CORS| raw:: html 
+
+      <span class="label label-success pull-right">
+        CORS
+      </span>

doc/sphinx-guides/source/api/search.rst

@@ -14,6 +14,8 @@ The parameters and JSON response are partly inspired by the `GitHub Search API <
 
 Please note that in Dataverse 4.3 and older the "citation" field wrapped the persistent ID URL in an ``<a>`` tag but this has been changed to plaintext. If you want the old value with HTML in it, a new field called "citationHtml" can be used.
 
+The search API can be used from scripts running in web browsers, as it allows cross-origin resource sharing (CORS).
+
 Parameters
 ----------
 

doc/sphinx-guides/source/developers/dev-environment.rst

@@ -150,17 +150,27 @@ The script is a variation of the old installer from DVN 3.x that calls another s
 
 All the future changes to the configuration that are Glassfish-specific and can be done through ``asadmin`` should now go into ``scripts/install/glassfish-setup.sh``.
 
-Shibboleth
-----------
+Shibboleth and OAuth
+--------------------
 
-If you are working on anything related to users, please keep in mind that your changes will likely affect Shibboleth users. Rather than setting up Shibboleth on your laptop, developers are advised to simply add a value to their database to enable Shibboleth "dev mode" like this:
+If you are working on anything related to users, please keep in mind that your changes will likely affect Shibboleth and OAuth users. Rather than setting up Shibboleth on your laptop, developers are advised to simply add a value to their database to enable Shibboleth "dev mode" like this:
 
 ``curl http://localhost:8080/api/admin/settings/:DebugShibAccountType -X PUT -d RANDOM``
 
 For a list of possible values, please "find usages" on the settings key above and look at the enum.
 
 Now when you go to http://localhost:8080/shib.xhtml you should be prompted to create a Shibboleth account.
 
+OAuth is much more straightforward to get working on your laptop than Shibboleth. GitHub is a good identity provider to test with because you can easily request a Client ID and Client Secret that works against localhost. Follow the instructions in the :doc:`/installation/oauth2` section of the installation Guide and use "http://localhost:8080/oauth2/callback.xhtml" as the callback URL.
+
+In addition to setting up OAuth on your laptop for real per above, you can also use a dev/debug mode:
+
+``curl http://localhost:8080/api/admin/settings/:DebugOAuthAccountType -X PUT -d RANDOM_EMAIL2``
+
+For a list of possible values, please "find usages" on the settings key above and look at the enum.
+
+Now when you go to http://localhost:8080/oauth2/firstLogin.xhtml you should be prompted to create a Shibboleth account.
+
 Rebuilding your dev environment
 -------------------------------
 

doc/sphinx-guides/source/installation/administration.rst

@@ -70,7 +70,7 @@ There is a database table called ``actionlogrecord`` that captures events that m
 Maintenance
 -----------
 
-When you have scheduled down time for your production servers, we provide a `sample maintenance page <../_static/installation/files/etc/maintenance/maintenance.xhtml>`_ for you to use. To download, right-click and select "Save Link As".
+When you have scheduled down time for your production servers, we provide a :download:`sample maintenance page <../_static/installation/files/etc/maintenance/maintenance.xhtml>` for you to use. To download, right-click and select "Save Link As".
 
 The maintenance page is intended to be a static page served by Apache to provide users with a nicer, more informative experience when the site is unavailable.
 
@@ -88,7 +88,7 @@ The app will send a standard welcome email with a URL the user can click, which,
 
 Should users' URL token expire, they will see a "Verify Email" button on the account information page to send another URL.
 
-Sysadmins can determine which users have verified their email addresses by looking for the presence of the value ``emailLastConfirmed`` in the JSON output from listing users (see the "Admin" section of the :doc:`/api/native-api`). The email addresses for Shibboleth users are re-confirmed on every login.
+Sysadmins can determine which users have verified their email addresses by looking for the presence of the value ``emailLastConfirmed`` in the JSON output from listing users (see the "Admin" section of the :doc:`/api/native-api`). As mentioned in the :doc:`/user/account` section of the User Guide, the email addresses for Shibboleth users are re-confirmed on every login.
 
 Deleting an API Token
 +++++++++++++++++++++