Merge pull request #5289 from poikilotherm/5288-dependency-housekeeping

5288 Introducing proper deps management and docs about it.

doc/sphinx-guides/source/conf.py

@@ -40,7 +40,8 @@
     'sphinx.ext.autodoc',
     'sphinx.ext.intersphinx',
     'sphinx.ext.ifconfig',
-    'sphinx.ext.viewcode'
+    'sphinx.ext.viewcode',
+    'sphinx.ext.graphviz'
 ]
 
 # Add any paths that contain templates here, relative to this directory.

doc/sphinx-guides/source/developers/dependencies.rst

@@ -0,0 +1,274 @@
+=====================
+Dependency Management
+=====================
+
+.. contents:: |toctitle|
+	:local:
+
+Dataverse is (currently) a Java EE 7 based application, that uses a lot of additional libraries for special purposes.
+This includes features like support for SWORD-API, S3 storage and many others.
+
+Besides the code that glues together the single pieces, any developer needs to describe used dependencies for the
+Maven-based build system. As is familiar to any Maven user, this happens inside the "Project Object Model" (POM) living in
+``pom.xml`` at the root of the project repository. Recursive and convergent dependency resolution makes dependency
+management with Maven very easy. But sometimes, in projects with many complex dependencies like Dataverse, you have
+to help Maven make the right choices.
+
+Terms
+-----
+
+As a developer, you should familiarize yourself with the following terms:
+
+- **Direct dependencies**: things *you use* yourself in your own code for Dataverse.
+- **Transitive dependencies**: things *others use* for things you use, pulled in recursively.
+  See also: `Maven docs <https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Transitive_Dependencies>`_.
+
+.. graphviz::
+
+    digraph {
+        rankdir="LR";
+        node [fontsize=10]
+
+        yc [label="Your Code"]
+        da [label="Direct Dependency A"]
+        db [label="Direct Dependency B"]
+        ta [label="Transitive Dependency TA"]
+        tb [label="Transitive Dependency TB"]
+        tc [label="Transitive Dependency TC"]
+        dtz [label="Direct/Transitive Dependency Z"]
+
+        yc -> da -> ta;
+        yc -> db -> tc;
+        da -> tb -> tc;
+        db -> dtz;
+        yc -> dtz;
+    }
+
+Direct dependencies
+-------------------
+
+Within the POM, any direct dependencies reside within the ``<dependencies>`` tag:
+
+.. code:: xml
+
+    <dependencies>
+        <dependency>
+            <groupId>org.example</groupId>
+            <artifactId>example</artifactId>
+            <version>1.1.0</version>
+            <scope>compile</scope>
+        </dependency>
+    </dependencies>
+
+
+Anytime you add a ``<dependency>``, Maven will try to fetch it from defined/configured repositories and use it
+within the build lifecycle. You have to define a ``<version>``, but ``<scope>`` is optional for ``compile``.
+(See `Maven docs: Dep. Scope <https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Scope>`_)
+
+
+During fetching, Maven will analyse all transitive dependencies (see graph above) and, if necessary, fetch those, too.
+Everything downloaded once is cached locally by default, so nothing needs to be fetched again and again, as long as the
+dependency definition does not change.
+
+**Rules to follow:**
+
+1. You should only use direct dependencies for **things you are actually using** in your code.
+2. **Clean up** direct dependencies no longer in use. It will bloat the deployment package otherwise!
+3. Care about the **scope**. Do not include "testing only" dependencies in the package - it will hurt you in IDEs and bloat things. [#f1]_
+4. Avoid using different dependencies for the **same purpose**, e. g. different JSON parsing libraries.
+5. Refactor your code to **use Java EE** standards as much as possible.
+6. When you rely on big SDKs or similar big cool stuff, try to **include the smallest portion possible**. Complete SDK
+   bundles are typically heavyweight and most of the time unnecessary.
+7. **Don't include transitive dependencies.** [#f2]_
+
+   * Exception: if you are relying on it in your code (see *Z* in the graph above), you must declare it. See below
+     for proper handling in these (rare) cases.
+
+
+Transitive dependencies
+-----------------------
+
+Maven is comfortable for developers; it handles recursive resolution, downloading, and adding "dependencies of dependencies".
+However, as life is a box of chocolates, you might find yourself in *version conflict hell* sooner than later without even
+knowing, but experiencing unintended side effects.
+
+When you look at the graph above, imagine *B* and *TB* rely on different *versions* of *TC*. How does Maven decide
+which version it will include? Easy: the dependent version of the nearest version wins:
+
+.. graphviz::
+
+    digraph {
+        rankdir="LR";
+        node [fontsize=10]
+
+        yc [label="Your Code"]
+        db [label="Direct Dependency B"]
+        dtz1 [label="Z v1.0"]
+        dtz2 [label="Z v2.0"]
+
+        yc -> db -> dtz1;
+        yc -> dtz2;
+    }
+
+In this case, version "2.0" will be included. If you know something about semantic versioning, a red alert should ring in your mind right now.
+How do we know that *B* is compatible with *Z v2.0* when depending on *Z v1.0*?
+
+Another scenario getting us in trouble: indirect use of transitive dependencies. Imagine the following: we rely on *Z*
+in our code, but do not include a direct dependency for it within the POM. Now *B* is updated and removed its dependency
+on *Z*. You definitely don't want to head down that road.
+
+**Follow the rules to be safe:**
+
+1. Do **not use transitive deps implicit**: add a direct dependency for transitive deps you re-use in your code.
+2. On every build check that no implicit usage was added by accident.
+3. **Explicitly declare versions** of transitive dependencies in use by multiple direct dependencies.
+4. On every build check that there are no convergence problems hiding in the shadows.
+5. **Do special tests** on every build to verify these explicit combinations work.
+
+Managing transitive dependencies in ``pom.xml``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Maven can manage versions of transitive dependencies in four ways:
+
+1. Make a transitive-only dependency not used in your code a direct one and add a ``<version>`` tag.
+   Typically a bad idea, don't do that.
+2. Use ``<optional>`` or ``<exclusion>`` tags on direct dependencies that request the transitive dependency.
+   *Last resort*, you really should avoid this. Not explained or used here.
+   `See Maven docs <https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html>`_.
+3. Explicitly declare the transitive dependency in ``<dependencyManagement>`` and add a ``<version>`` tag.
+4. For more complex transitive dependencies, reuse a "Bill of Materials" (BOM) within ``<dependencyManagement>``
+   and add a ``<version>`` tag. Many bigger and standard use projects provide those, making the POM much less bloated
+   compared to adding every bit yourself.
+
+A reduced example, only showing bits relevant to the above cases and usage of an explicit transitive dep directly:
+
+.. code-block:: xml
+    :linenos:
+
+    <properties>
+        <aws.version>1.11.172</aws.version>
+        <!-- We need to ensure that our choosen version is compatible with every dependency relying on it.
+             This is manual work and needs testing, but a good investment in stability and up-to-date dependencies. -->
+        <jackson.version>2.9.6</jackson.version>
+        <joda.version>2.10.1</joda.version>
+    </properties>
+
+    <!-- Transitive dependencies, bigger library "bill of materials" (BOM) and
+         versions of dependencies used both directly and transitive are managed here. -->
+    <dependencyManagement>
+        <dependencies>
+            <!-- First example for case 4. Only one part of the SDK (S3) is used and transitive deps
+                 of that are again managed by the upstream BOM. -->
+            <dependency>
+                <groupId>com.amazonaws</groupId>
+                <artifactId>aws-java-sdk-bom</artifactId>
+                <version>${aws.version}</version>
+                <type>pom</type>
+                <scope>import</scope>
+            </dependency>
+            <!-- Second example for case 4 and an example for explicit direct usage of a transitive dependency.
+                 Jackson is used by AWS SDK and others, but we also use it in Dataverse. -->
+            <dependency>
+                <groupId>com.fasterxml.jackson</groupId>
+                <artifactId>jackson-bom</artifactId>
+                <version>${jackson.version}</version>
+                <scope>import</scope>
+                <type>pom</type>
+            </dependency>
+            <!-- Example for case 3. Joda is not used in Dataverse (as of writing this). -->
+            <dependency>
+                <groupId>joda-time</groupId>
+                <artifactId>joda-time</artifactId>
+                <version>${joda.version}</version>
+            </dependency>
+        </dependencies>
+    </dependencyManagement>
+
+    <!-- Declare any DIRECT dependencies here.
+         In case the depency is both transitive and direct (e. g. some common lib for logging),
+         manage the version above and add the direct dependency here WITHOUT version tag, too.
+    -->
+    <dependencies>
+        <dependency>
+            <groupId>com.amazonaws</groupId>
+            <artifactId>aws-java-sdk-s3</artifactId>
+            <!-- no version here as managed by BOM above! -->
+        </dependency>
+        <!-- Should be refactored and removed once on Java EE 8 -->
+        <dependency>
+            <groupId>com.fasterxml.jackson.core</groupId>
+            <artifactId>jackson-core</artifactId>
+            <!-- no version here as managed above! -->
+        </dependency>
+        <!-- Should be refactored and removed once on Java EE 8 -->
+        <dependency>
+            <groupId>com.fasterxml.jackson.core</groupId>
+            <artifactId>jackson-databind</artifactId>
+            <!-- no version here as managed above! -->
+        </dependency>
+    </dependencies>
+
+
+Helpful tools
+--------------
+
+Maven provides some plugins that are of great help to detect possible conflicts and implicit usage.
+
+For *implicit usage detection*, use `mvn dependency:analyze`. Examine the output with great care. Sometimes you will
+see implicit usages that do no harm, especially if you are using bigger SDKs having some kind of `core` package.
+This will also report on any direct dependency which is not in use and can be removed from the POM. Again, do this with
+great caution and double check.
+
+If you want to see the dependencies both direct and transitive in a *dependency tree format*, use `mvn dependency:tree`.
+
+This will however not help you with detecting possible version conflicts. For this you need to use the `Enforcer Plugin
+<https://maven.apache.org/enforcer/maven-enforcer-plugin/index.html>`_ with its built in `dependency convergence rule
+<https://maven.apache.org/enforcer/enforcer-rules/dependencyConvergence.html>`_. 
+
+Repositories
+------------
+
+Maven receives all dependencies from *repositories*. Those can be public like `Maven Central <https://search.maven.org/>`_
+and others, but you can also use a private repository on premises or in the cloud. Last but not least, you can use
+local repositories, which can live next to your application code (see ``local_lib`` dir within Dataverse codebase).
+
+Repositories are defined within the Dataverse POM like this:
+
+.. code:: xml
+
+    <repositories>
+        <repository>
+            <id>central-repo</id>
+            <name>Central Repository</name>
+            <url>http://repo1.maven.org/maven2</url>
+            <layout>default</layout>
+        </repository>
+        <repository>
+            <id>prime-repo</id>
+            <name>PrimeFaces Maven Repository</name>
+            <url>http://repository.primefaces.org</url>
+            <layout>default</layout>
+        </repository>
+        <repository>
+            <id>dvn.private</id>
+            <name>Local repository for hosting jars not available from network repositories.</name>
+            <url>file://${project.basedir}/local_lib</url>
+        </repository>
+    </repositories>
+
+You can also add repositories to your local Maven settings, see `docs <https://maven.apache.org/ref/3.6.0/maven-settings/settings.html>`_.
+
+Typically you will skip the addition of the central repository, but adding it to the POM has the benefit that
+dependencies are first looked up there (which in theory can speed up downloads). You should keep in mind that repositories
+are used in the order they appear.
+
+----
+
+.. rubric:: Footnotes
+
+.. [#f1] Modern IDEs import your Maven POM and offer import autocompletion for classes based on direct dependencies in the model. You might end up using legacy or repackaged classes because of a wrong scope.
+.. [#f2] This is going to bite back in modern IDEs when importing classes from transitive dependencies by "autocompletion accident".
+
+----
+
+Previous: :doc:`documentation` | Next: :doc:`debugging`

doc/sphinx-guides/source/developers/documentation.rst

@@ -73,6 +73,17 @@ Every non-index page should use the following code to display a table of content
 
 This code should be placed below any introductory text/images and directly above the first subheading, much like a Wikipedia page.
 
+GraphViz based images
+---------------------
+
+In some parts of the documentation, graphs are rendered as images via Sphinx GraphViz extension.
+
+This requires `GraphViz <http://graphviz.org/>`_ installed and either ``dot`` on the path or
+`adding options to the make call <https://groups.google.com/forum/#!topic/sphinx-users/yXgNey_0M3I>`_.
+
+This has been tested and works on Mac, Linux, and Windows. If you have not properly configured GraphViz, then the worst thing that might happen is a warning and missing images in your local documentation build.
+
+
 Versions
 --------
 
@@ -86,4 +97,4 @@ In order to make it clear to the crawlers that we only want the latest version d
 
 ----
 
-Previous: :doc:`testing` | Next: :doc:`debugging`
+Previous: :doc:`testing` | Next: :doc:`dependencies`

doc/sphinx-guides/source/developers/index.rst

@@ -19,6 +19,7 @@ Developer Guide
    sql-upgrade-scripts
    testing
    documentation
+   dependencies
    debugging
    coding-style
    deployment

local_lib/com/lyncode/xoai-common/4.1.0-header-patch/xoai-common-4.1.0-header-patch.jar.md5

@@ -0,0 +1 @@
+f578d8ec91811d5d72981355cb7a1f0f

local_lib/com/lyncode/xoai-common/4.1.0-header-patch/xoai-common-4.1.0-header-patch.jar.sha1

@@ -0,0 +1 @@
+523abaf48b4423eb874dbc086b876aa917930a04

local_lib/com/lyncode/xoai-common/4.1.0-header-patch/xoai-common-4.1.0-header-patch.pom.md5

@@ -0,0 +1 @@
+346e9f235523e52256006bbe8eba60bb

local_lib/com/lyncode/xoai-common/4.1.0-header-patch/xoai-common-4.1.0-header-patch.pom.sha1

@@ -0,0 +1 @@
+cd83d08c097d6aa1b27b20ef4742c7e4fa47e6b5

local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch-javadoc.jar.md5

@@ -0,0 +1 @@
+546f1ab3f3f654280f88e429ba3471ae

local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch-javadoc.jar.sha1

@@ -0,0 +1 @@
+f4da7ebc3fda69e1e7db12bda6d7b5fb4aecc7a4

local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch-sources.jar.md5

@@ -0,0 +1 @@
+ec87ba7cb8e7396fc903acdbacd31ff6

local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch-sources.jar.sha1

@@ -0,0 +1 @@
+370c2955550a42b11fe7b9007771c506f5769639

local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch.jar.md5

@@ -0,0 +1 @@
+544e9b97062d054370695b9b09d4bb1c

local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch.jar.sha1

@@ -0,0 +1 @@
+67c505461f3c190894bb036cc866eb640c2f6a48

local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch.pom.md5

@@ -0,0 +1 @@
+a1b49c13fcf448de9628798f8682fcaa

local_lib/com/lyncode/xoai-data-provider/4.1.0-header-patch/xoai-data-provider-4.1.0-header-patch.pom.sha1

@@ -0,0 +1 @@
+41be98af31f8d17d83ab6c38bd7939ba212eab8d

local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch-javadoc.jar.md5

@@ -0,0 +1 @@
+21bc45a29b715720f4b77f51bf9f1754

local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch-javadoc.jar.sha1

@@ -0,0 +1 @@
+b544162e82d322116b87d99f2fbb6ddd4c4745e1

local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch-sources.jar.md5

@@ -0,0 +1 @@
+88dc05805672ebe01ded1197a582cd60

local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch-sources.jar.sha1

@@ -0,0 +1 @@
+fe41289cb74c56e9282dd09c22df2eda47c68a0d

local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch.jar.md5

@@ -0,0 +1 @@
+52f8b446f78009757d593312778f428c

local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch.jar.sha1

@@ -0,0 +1 @@
+feb6903ad32d4b42461b7ca1b3fae6146740bb31

local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch.pom.md5

@@ -0,0 +1 @@
+b97b8ee92daa5fc4fd87004465f9ad2b

local_lib/com/lyncode/xoai-service-provider/4.1.0-header-patch/xoai-service-provider-4.1.0-header-patch.pom.sha1

@@ -0,0 +1 @@
+f772583549263bd72ea4d5268d9db0a84c27cb9f

local_lib/com/lyncode/xoai/4.1.0-header-patch/xoai-4.1.0-header-patch.pom.md5

@@ -0,0 +1 @@
+b50966bebe8cfdcb58478cf029b08aa3

local_lib/com/lyncode/xoai/4.1.0-header-patch/xoai-4.1.0-header-patch.pom.sha1

@@ -0,0 +1 @@
+28a5d65399cbc25b29b270caebbb86e292c5ba18

local_lib/edu/harvard/hul/ois/jhove/jhove-handler/1.11.0/jhove-handler-1.11.0.jar.md5

@@ -0,0 +1 @@
+f9bb7a20a9d538819606ec1630d661fe

local_lib/edu/harvard/hul/ois/jhove/jhove-handler/1.11.0/jhove-handler-1.11.0.jar.sha1

@@ -0,0 +1 @@
+37a9d8e464a57b90c04252f265572e5274beb605

local_lib/edu/harvard/hul/ois/jhove/jhove-handler/1.11.0/jhove-handler-1.11.0.pom.md5

@@ -0,0 +1 @@
+c2d1a458dc809cb3833f3b362a23ed79

local_lib/edu/harvard/hul/ois/jhove/jhove-handler/1.11.0/jhove-handler-1.11.0.pom.sha1

@@ -0,0 +1 @@
+0f195ee47691c7ee8611db63b6d5ee262c139129

local_lib/edu/harvard/hul/ois/jhove/jhove-module/1.11.0/jhove-module-1.11.0.jar.md5

@@ -0,0 +1 @@
+c3605bd6434ebeef82ef655d21075652

local_lib/edu/harvard/hul/ois/jhove/jhove-module/1.11.0/jhove-module-1.11.0.jar.sha1

@@ -0,0 +1 @@
+d8dc496b4d408dd6a9ed7429e6fa4d1ce5f57403

local_lib/edu/harvard/hul/ois/jhove/jhove-module/1.11.0/jhove-module-1.11.0.pom.md5

@@ -0,0 +1 @@
+bcac19fbdf825c5e93e785413815b998

local_lib/edu/harvard/hul/ois/jhove/jhove-module/1.11.0/jhove-module-1.11.0.pom.sha1

@@ -0,0 +1 @@
+1f983c8cf895056f4d4efe7a717b8d73d5c6b091

local_lib/edu/harvard/hul/ois/jhove/jhove/1.11.0/jhove-1.11.0.jar.md5

@@ -0,0 +1 @@
+3f6f413fb54c5142f2e34837bb9369b4

local_lib/edu/harvard/hul/ois/jhove/jhove/1.11.0/jhove-1.11.0.jar.sha1

@@ -0,0 +1 @@
+475409b6444aba6bdc96ce42431b6d601c7abe5f

local_lib/edu/harvard/hul/ois/jhove/jhove/1.11.0/jhove-1.11.0.pom.md5

@@ -0,0 +1 @@
+7f9939585e369ad60ac1f8a99b2fa75f

local_lib/edu/harvard/hul/ois/jhove/jhove/1.11.0/jhove-1.11.0.pom.sha1

@@ -0,0 +1 @@
+804fffb163526c6bea975038702ea90f24f89419

local_lib/edu/harvard/iq/dvn/unf5/5.0/unf5-5.0.jar.md5

@@ -0,0 +1 @@
+eeef5c0dc201d1105b9529a51fa8cdab

local_lib/edu/harvard/iq/dvn/unf5/5.0/unf5-5.0.jar.sha1

@@ -0,0 +1 @@
+1fa716d318920fd59fc63f77965d113decf97355

local_lib/edu/harvard/iq/dvn/unf5/5.0/unf5-5.0.pom.md5

@@ -0,0 +1 @@
+2df5dac09375e1e7fcd66c705d9ca2ef

local_lib/edu/harvard/iq/dvn/unf5/5.0/unf5-5.0.pom.sha1

@@ -0,0 +1 @@
+431cd55e2e9379677d14e402dd3c474bb7be4ac9

local_lib/net/handle/handle/2006-06-16-generated/handle-2006-06-16-generated.jar.md5

@@ -0,0 +1 @@
+f6099186cd4ef67ea91b4c3b724c1113

local_lib/net/handle/handle/2006-06-16-generated/handle-2006-06-16-generated.jar.sha1

@@ -0,0 +1 @@
+2232318434cab52dd755fba7003958204459f404

local_lib/net/handle/handle/2006-06-16-generated/handle-2006-06-16-generated.pom.md5

@@ -0,0 +1 @@
+b1390a875687dad3cc6527b83f84e635