Skip to content

Commit

Permalink
Merge branch 'develop' into 4158-update-solr
Browse files Browse the repository at this point in the history
  • Loading branch information
matthew-a-dunlap committed Mar 14, 2018
2 parents 8fef67b + 5ed5edf commit f8020cd
Show file tree
Hide file tree
Showing 265 changed files with 9,208 additions and 2,126 deletions.
8 changes: 8 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -33,3 +33,11 @@ scripts/api/setup-all.sh*

# ctags generated tag file
tags

# dependencies I'm not sure we're allowed to redistribute / have in version control
conf/docker-aio/dv/deps/

# no need to check aoi installer zip into vc
conf/docker-aio/dv/install/dvinstall.zip
# or copy of test data
conf/docker-aio/testdata/
28 changes: 16 additions & 12 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,19 @@
# Contributing to Dataverse

Thank you for your interest in contributing to Dataverse! We are open to contributions from everyone. You don't need permission to participate, just jump in using the resources below. If you have questions, reach out to us on the [#dataverse IRC channel][], and hang around a while, as it may take time for community members to de-idle.
Thank you for your interest in contributing to Dataverse! We are open to contributions from everyone. You don't need permission to participate. Just jump in. If you have questions, please reach out using one or more of the channels described below.

We aren't just looking for developers, there are many ways to contribute to Dataverse. We welcome contributions of ideas, bug reports, usability research/feedback, documentation, code, and more!
We aren't just looking for developers. There are many ways to contribute to Dataverse. We welcome contributions of ideas, bug reports, usability research/feedback, documentation, code, and more!

## Ideas/Feature Requests

Your idea or feature request might already be captured in the Dataverse [issue tracker] on GitHub but if not, the best way to bring it to the community's attention is by posting on the [dataverse-community Google Group][]. You're also welcome make some noise in the [#dataverse IRC channel][] (which is [logged][]) or cram your idea into 140 characters and mention [@dataverseorg][] on Twitter. To discuss your idea privately, please email it to support@dataverse.org
Your idea or feature request might already be captured in the Dataverse [issue tracker] on GitHub but if not, the best way to bring it to the community's attention is by posting on the [dataverse-community Google Group][] or bringing it up on a [Community Call][]. You're also welcome make some noise in the [#dataverse IRC channel][] (which is [logged][]) or cram your idea into 280 characters and mention [@dataverseorg][] on Twitter. To discuss your idea privately, please email it to support@dataverse.org

There's a chance your idea is already on our roadmap, which is available at http://dataverse.org/goals-roadmap-and-releases

[#dataverse IRC channel]: http://chat.dataverse.org
[logged]: http://irclog.iq.harvard.edu/dataverse/today
[issue tracker]: https://github.com/IQSS/dataverse/issues
[@dataverseorg]: https://twitter.com/dataverseorg
[Functional Requirements Document (FRD for short)]: https://docs.google.com/document/d/1PRyAlP6zlUlUuHfgyUezzuaVQ4JnapvgtGWo0o7tLEs/edit?usp=sharing
[Balsamiq]: https://iqssharvard.mybalsamiq.com/projects
[Functional Requirements Document folder on Google Drive]: https://drive.google.com/folderview?id=0B3_V6vFxEcx-fl92ek92OG1nTmhQenBRX1Z4OVJBLXpURmh2d2RyX1NZRUp6YktaYUU5YTA&usp=sharing

## Usability testing

Expand All @@ -26,15 +23,15 @@ Please email us at support@dataverse.org if you are interested in participating

An issue is a bug (a feature is no longer behaving the way it should) or a feature (something new to Dataverse that helps users complete tasks). You can browse the Dataverse [issue tracker] on GitHub by open or closed issues or by milestones.

Before submitting an issue, please search the existing issues by using the search bar at the top of the page. If there is an existing issue that matches the issue you want to report, please add a comment to it.
Before submitting an issue, please search the existing issues by using the search bar at the top of the page. If there is an existing open issue that matches the issue you want to report, please add a comment to it.

If there is no pre-existing issue, please click on the "New Issue" button, log in, and write in what the issue is (unless it is a security issue which should be reported privately to security@dataverse.org).
If there is no pre-existing issue or it has been closed, please click on the "New Issue" button, log in, and write in what the issue is (unless it is a security issue which should be reported privately to security@dataverse.org).

If you do not receive a reply to your new issue or comment in a timely manner, please email support@dataverse.org with a link to the issue.

### Writing an Issue

For the subject of an issue, please start it by writing the feature or functionality it relates to, i.e. "Create Account:..." or "Dataset Page:...". In the body of the issue, please outline the issue you are reporting with as much detail as possible. In order for the Dataverse development team to best respond to the issue, we need as much information about the issue as you can provide. Include steps to reproduce bugs. Indicate which version you're using. We love screenshots!
For the subject of an issue, please start it by writing the feature or functionality it relates to, i.e. "Create Account:..." or "Dataset Page:...". In the body of the issue, please outline the issue you are reporting with as much detail as possible. In order for the Dataverse development team to best respond to the issue, we need as much information about the issue as you can provide. Include steps to reproduce bugs. Indicate which version you're using, which is shown at the bottom of the page. We love screenshots!

### Issue Attachments

Expand All @@ -51,13 +48,20 @@ The source for the documentation at http://guides.dataverse.org/en/latest/ is in

## Code/Pull Requests

Before you start coding, please reach out to us either on the [dataverse-community Google Group][], the [dataverse-dev Google Group][], [IRC][] (#dataverse on freenode), or via support@dataverse.org to make sure the effort is well coordinated and we avoid merge conflicts.
We love code contributions. Developers are not limited to the main Dataverse code in this git repo. You can help with API client libraries in your favorite language that are mentioned in the [API Guide][] or create a new library. You can help work on configuration management code that's mentioned in the [Installation Guide][]. The Installation Guide also covers a new concept called "external tools" that allows developers to create their own tools that are available from within an installation of Dataverse.

Please read http://guides.dataverse.org/en/latest/developers/version-control.html to understand how we use the "git flow" model of development and how we will encourage you to create a GitHub issue (if it doesn't exist already) to associate with your pull request.
[API Guide]: http://guides.dataverse.org/en/latest/api
[Installation Guide]: http://guides.dataverse.org/en/latest/installation

After making your pull request, your goal should be to help it advance through our kanban board at https://waffle.io/IQSS/dataverse . If no one has moved your pull request to the code review column in a timely manner, please reach out. We maintain a list of [community contributors][] so please let us know if you'd like to be added or removed from the list. Thanks!
If you are interested in working on the main Dataverse code, great! Before you start coding, please reach out to us either on the [dataverse-community Google Group][], the [dataverse-dev Google Group][], [IRC][] (#dataverse on freenode), or via support@dataverse.org to make sure the effort is well coordinated and we avoid merge conflicts. We maintain a list of [community contributors][] and [dev efforts][] the community is working on so please let us know if you'd like to be added or removed from either list.

Please read http://guides.dataverse.org/en/latest/developers/version-control.html to understand how we use the "git flow" model of development and how we will encourage you to create a GitHub issue (if it doesn't exist already) to associate with your pull request. That page also includes tips on making a pull request.

After making your pull request, your goal should be to help it advance through our kanban board at https://waffle.io/IQSS/dataverse . If no one has moved your pull request to the code review column in a timely manner, please reach out. Thanks!

[dataverse-community Google Group]: https://groups.google.com/group/dataverse-community
[Community Call]: https://dataverse.org/community-calls
[dataverse-dev Google Group]: https://groups.google.com/group/dataverse-dev
[IRC]: http://chat.dataverse.org
[community contributors]: https://docs.google.com/spreadsheets/d/1o9DD-MQ0WkrYaEFTD5rF_NtyL8aUISgURsAXSL7Budk/edit?usp=sharing
[dev efforts]: https://groups.google.com/d/msg/dataverse-community/X2diSWYll0w/ikp1TGcfBgAJ
1 change: 1 addition & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
# See `conf/docker` for Docker images
13 changes: 4 additions & 9 deletions Vagrantfile
Original file line number Diff line number Diff line change
Expand Up @@ -8,18 +8,13 @@ Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

config.vm.define "standalone", primary: true do |standalone|
config.vm.hostname = "standalone"
standalone.vm.box = "puppet-vagrant-boxes.puppetlabs.com-centos-65-x64-virtualbox-puppet.box"
# Uncomment this temporarily to get `vagrant destroy` to work
#standalone.vm.box = "puppetlabs/centos-7.2-64-puppet"

operating_system = "centos"
if ENV['OPERATING_SYSTEM'].nil?
puts "OPERATING_SYSTEM environment variable not specified. Using #{operating_system} by default.\nTo specify it in bash: export OPERATING_SYSTEM=debian"
config.vm.box_url = "http://puppet-vagrant-boxes.puppetlabs.com/centos-65-x64-virtualbox-puppet.box"
config.vm.box = "puppet-vagrant-boxes.puppetlabs.com-centos-65-x64-virtualbox-puppet.box"
elsif ENV['OPERATING_SYSTEM'] == 'centos7'
puts "WARNING: CentOS 7 specified. Newer than what the dev team tests on."
config.vm.box_url = "https://atlas.hashicorp.com/puppetlabs/boxes/centos-7.2-64-puppet/versions/1.0.1/providers/virtualbox.box"
config.vm.box = "puppetlabs-centos-7.2-64-puppet-1.0.1-virtualbox.box"
standalone.vm.box = "puppetlabs-centos-7.2-64-puppet-1.0.1-virtualbox.box"
config.vm.box = "puppetlabs/centos-7.2-64-puppet"
config.vm.box_version = '1.0.1'
elsif ENV['OPERATING_SYSTEM'] == 'debian'
puts "WARNING: Debian specified. Here be dragons! https://github.com/IQSS/dataverse/issues/1059"
config.vm.box_url = "http://puppet-vagrant-boxes.puppetlabs.com/debian-73-x64-virtualbox-puppet.box"
Expand Down
27 changes: 27 additions & 0 deletions conf/docker-aio/0prep_deps.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
#!/bin/sh
if [ ! -d dv/deps ]; then
mkdir -p dv/deps
fi
wdir=`pwd`
if [ ! -e dv/deps/glassfish4dv.tgz ]; then
echo "glassfish dependency prep"
mkdir -p /tmp/dv-prep/gf
cd /tmp/dv-prep/gf
wget http://download.java.net/glassfish/4.1/release/glassfish-4.1.zip
wget http://search.maven.org/remotecontent?filepath=org/jboss/weld/weld-osgi-bundle/2.2.10.Final/weld-osgi-bundle-2.2.10.Final-glassfish4.jar -O weld-osgi-bundle-2.2.10.Final-glassfish4.jar
unzip glassfish-4.1.zip
rm glassfish4/glassfish/modules/weld-osgi-bundle.jar
mv weld-osgi-bundle-2.2.10.Final-glassfish4.jar glassfish4/glassfish/modules
tar zcf $wdir/dv/deps/glassfish4dv.tgz glassfish4
cd $wdir
# assuming that folks usually have /tmp auto-clean as needed
fi

if [ ! -e dv/deps/solr-4.6.0dv.tgz ]; then
echo "solr dependency prep"
# schema changes *should* be the only ones...
cd dv/deps/
wget https://archive.apache.org/dist/lucene/solr/4.6.0/solr-4.6.0.tgz -O solr-4.6.0dv.tgz
cd ../../
fi

24 changes: 24 additions & 0 deletions conf/docker-aio/1prep.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
#!/bin/sh

export LANG="en_US.UTF-8"

# move things necessary for integration tests into build context.
# this was based off the phoenix deployment; and is likely uglier and bulkier than necessary in a perfect world

mkdir -p testdata/doc/sphinx-guides/source/_static/util/
cp ../solr/4.6.0/schema.xml testdata/
cp ../jhove/jhove.conf testdata/
cd ../../
cp -r scripts conf/docker-aio/testdata/
cp doc/sphinx-guides/source/_static/util/pg8-createsequence-prep.sql conf/docker-aio/testdata/doc/sphinx-guides/source/_static/util/
cp doc/sphinx-guides/source/_static/util/createsequence.sql conf/docker-aio/testdata/doc/sphinx-guides/source/_static/util/

# not using dvinstall.zip for setupIT.bash; but still used in install.bash for normal ops
mvn clean
./scripts/database/homebrew/custom-build-number
mvn package
cd scripts/installer
make clean
make
cp dvinstall.zip ../../conf/docker-aio/dv/install/

40 changes: 40 additions & 0 deletions conf/docker-aio/c7.dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
FROM centos:7
# OS dependencies
RUN yum install -y java-1.8.0-openjdk-headless postgresql-server sudo epel-release unzip perl curl
RUN yum install -y jq

# copy and unpack dependencies (solr, glassfish)
COPY dv /tmp/dv
COPY testdata/schema.xml /tmp/dv
RUN cd /opt ; tar zxf /tmp/dv/deps/solr-4.6.0dv.tgz
RUN cd /opt ; tar zxf /tmp/dv/deps/glassfish4dv.tgz

RUN sudo -u postgres /usr/bin/initdb -D /var/lib/pgsql/data
#RUN sudo -u postgres createuser dvnapp

# copy configuration related files
RUN cp /tmp/dv/pg_hba.conf /var/lib/pgsql/data/ ; cp /tmp/dv/schema.xml /opt/solr-4.6.0/example/solr/collection1/conf/schema.xml

# skipping glassfish user and solr user (run both as root)

#solr port
EXPOSE 8983

# postgres port
EXPOSE 5432

# glassfish port
EXPOSE 8080

RUN mkdir /opt/dv

# yeah - still not happy if glassfish isn't in /usr/local :<
RUN ln -s /opt/glassfish4 /usr/local/glassfish4
COPY dv/install/ /opt/dv/
COPY install.bash /opt/dv/
COPY entrypoint.bash /opt/dv/
COPY testdata /opt/dv/testdata
COPY testscripts/* /opt/dv/testdata/
COPY setupIT.bash /opt/dv
WORKDIR /opt/dv
CMD ["/opt/dv/entrypoint.bash"]
16 changes: 16 additions & 0 deletions conf/docker-aio/default.config
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
HOST_DNS_ADDRESS localhost
GLASSFISH_DIRECTORY /opt/glassfish4
ADMIN_EMAIL
MAIL_SERVER mail.hmdc.harvard.edu
POSTGRES_ADMIN_PASSWORD secret
POSTGRES_SERVER db
POSTGRES_PORT 5432
POSTGRES_DATABASE dvndb
POSTGRES_USER dvnapp
POSTGRES_PASSWORD secret
SOLR_LOCATION idx
TWORAVENS_LOCATION NOT INSTALLED
RSERVE_HOST localhost
RSERVE_PORT 6311
RSERVE_USER rserve
RSERVE_PASSWORD rserve
16 changes: 16 additions & 0 deletions conf/docker-aio/dv/install/default.config
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
HOST_DNS_ADDRESS localhost
GLASSFISH_DIRECTORY /opt/glassfish4
ADMIN_EMAIL
MAIL_SERVER mail.hmdc.harvard.edu
POSTGRES_ADMIN_PASSWORD secret
POSTGRES_SERVER db
POSTGRES_PORT 5432
POSTGRES_DATABASE dvndb
POSTGRES_USER dvnapp
POSTGRES_PASSWORD secret
SOLR_LOCATION idx
TWORAVENS_LOCATION NOT INSTALLED
RSERVE_HOST localhost
RSERVE_PORT 6311
RSERVE_USER rserve
RSERVE_PASSWORD rserve
91 changes: 91 additions & 0 deletions conf/docker-aio/dv/pg_hba.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# PostgreSQL Client Authentication Configuration File
# ===================================================
#
# Refer to the "Client Authentication" section in the PostgreSQL
# documentation for a complete description of this file. A short
# synopsis follows.
#
# This file controls: which hosts are allowed to connect, how clients
# are authenticated, which PostgreSQL user names they can use, which
# databases they can access. Records take one of these forms:
#
# local DATABASE USER METHOD [OPTIONS]
# host DATABASE USER ADDRESS METHOD [OPTIONS]
# hostssl DATABASE USER ADDRESS METHOD [OPTIONS]
# hostnossl DATABASE USER ADDRESS METHOD [OPTIONS]
#
# (The uppercase items must be replaced by actual values.)
#
# The first field is the connection type: "local" is a Unix-domain
# socket, "host" is either a plain or SSL-encrypted TCP/IP socket,
# "hostssl" is an SSL-encrypted TCP/IP socket, and "hostnossl" is a
# plain TCP/IP socket.
#
# DATABASE can be "all", "sameuser", "samerole", "replication", a
# database name, or a comma-separated list thereof. The "all"
# keyword does not match "replication". Access to replication
# must be enabled in a separate record (see example below).
#
# USER can be "all", a user name, a group name prefixed with "+", or a
# comma-separated list thereof. In both the DATABASE and USER fields
# you can also write a file name prefixed with "@" to include names
# from a separate file.
#
# ADDRESS specifies the set of hosts the record matches. It can be a
# host name, or it is made up of an IP address and a CIDR mask that is
# an integer (between 0 and 32 (IPv4) or 128 (IPv6) inclusive) that
# specifies the number of significant bits in the mask. A host name
# that starts with a dot (.) matches a suffix of the actual host name.
# Alternatively, you can write an IP address and netmask in separate
# columns to specify the set of hosts. Instead of a CIDR-address, you
# can write "samehost" to match any of the server's own IP addresses,
# or "samenet" to match any address in any subnet that the server is
# directly connected to.
#
# METHOD can be "trust", "reject", "md5", "password", "gss", "sspi",
# "krb5", "ident", "peer", "pam", "ldap", "radius" or "cert". Note that
# "password" sends passwords in clear text; "md5" is preferred since
# it sends encrypted passwords.
#
# OPTIONS are a set of options for the authentication in the format
# NAME=VALUE. The available options depend on the different
# authentication methods -- refer to the "Client Authentication"
# section in the documentation for a list of which options are
# available for which authentication methods.
#
# Database and user names containing spaces, commas, quotes and other
# special characters must be quoted. Quoting one of the keywords
# "all", "sameuser", "samerole" or "replication" makes the name lose
# its special character, and just match a database or username with
# that name.
#
# This file is read on server startup and when the postmaster receives
# a SIGHUP signal. If you edit the file on a running system, you have
# to SIGHUP the postmaster for the changes to take effect. You can
# use "pg_ctl reload" to do that.

# Put your actual configuration here
# ----------------------------------
#
# If you want to allow non-local connections, you need to add more
# "host" records. In that case you will also need to make PostgreSQL
# listen on a non-local interface via the listen_addresses
# configuration parameter, or via the -i or -h command line switches.



# TYPE DATABASE USER ADDRESS METHOD

# "local" is for Unix domain socket connections only
#local all all peer
local all all trust
# IPv4 local connections:
#host all all 127.0.0.1/32 trust
host all all 0.0.0.0/0 trust
# IPv6 local connections:
host all all ::1/128 trust
# Allow replication connections from localhost, by a user with the
# replication privilege.
#local replication postgres peer
#host replication postgres 127.0.0.1/32 ident
#host replication postgres ::1/128 ident
10 changes: 10 additions & 0 deletions conf/docker-aio/entrypoint.bash
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
#!/usr/bin/env bash

sudo -u postgres /usr/bin/postgres -D /var/lib/pgsql/data &
cd /opt/solr-4.6.0/example/
java -DSTOP.PORT=8079 -DSTOP.KEY=a09df7a0d -jar start.jar &

cd /opt/glassfish4
bin/asadmin start-domain
sleep infinity

9 changes: 9 additions & 0 deletions conf/docker-aio/install.bash
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
#!/usr/bin/env bash
sudo -u postgres createuser --superuser dvnapp
#./entrypoint.bash &
unzip dvinstall.zip
cd dvinstall/
./install -admin_email=pameyer+dvinstall@crystal.harvard.edu -y -f > install.out 2> install.err

echo "installer complete"
cat install.err
Loading

0 comments on commit f8020cd

Please sign in to comment.