Skip to content

AddonDepgraph

Jump to bottom Edit New page
John Casey edited this page Aug 20, 2013 · 21 revisions

The Depgraph add-on is by far the most extensive so far. It provides no fewer than nine REST resource classes, and the list keeps growing. As of the 0.8.1 release, Depgraph can:

Overview of REST Resources

  • Manage Atlas DB workspaces
    • Workspaces provide a durable configuration of profiles and source URIs allowed in depgraph results, plus saved selections for snapshots, version ranges, and similar
    • Basic CRUD operations, plus addition of activated profiles and allowed source URIs
  • Resolve dependency graphs
  • Add or query metadata attached to GAVs
  • Query for particular relationships on either the project (direct) or graph (indirect OR direct) level
  • Calculate things like intersection, addition, subtraction, or simple diff between two graphs (or even filtered views of the same graph)
  • Generate specific output formats of the relationships contained within a depgraph
    • BOMs
    • Graphviz files
    • dependency:tree -like output
    • Raw JSON
    • JSON mapping of GAV to list-of-urls with options to configure the artifact attachments to include in the list
    • Dense mock-up of the Maven console output's Downloading: ... lines, which are consumed by some tools

Core Event Listeners

Depgraph also has event listeners that will respond to proxied POM metadata, adding the parsed relationships to the Atlas DB passively.

General Workflow

In general, Depgraph requires a fairly complex interaction carried over multiple REST calls:

  1. Establish a new workspace:

    $ curl -X POST http://localhost:8080/aprox/api/1.0/depgraph/ws/new
    {"config":{"forceVersions":true},"selectedVersions":{},"wildcardSelectedVersions":{},"id":"409","open":true,"lastAccess":1376968924979}

  2. Configure the workspace with active profiles, etc.:

    $ curl -X PUT http://localhost:8080/aprox/api/1.0/depgraph/ws/409/profile/foo
    {
  "config": {
    "activePomLocations": ["pom:profile:foo", "pom:root"],
    "activeSources": ["unknown:unknown"],
    "forceVersions": true
  },
  "selectedVersions": {},
  "wildcardSelectedVersions": {},
  "id": "409",
  "open": true,
  "lastAccess": 1376969003663
}

  3. Discover the dependency graph for a project:

    $ curl 'http://localhost:8080/aprox/api/1.0/depgraph/resolve/group:public/org.commonjava.aprox.wars/aprox-savant/0.8.1?wsid=409&preset=sob'
    {"resolvedTopLevelGAVs": ["org.commonjava.aprox.wars:aprox-savant:0.8.1"]}

    Notice that I'm using a preset named sob. This is the default, but I've included it to illustrate the preset feature, which sets up a hard-coded graph filter for some common task. In this case, 'sob' stands for Shipping-Oriented Builds. In other words, it includes all the runtime artifacts you might want to rebuild yourself, plus directly-referenced test- and provided-scope dependencies, which you'll need in order to run your builds. Beyond that, it also includes all referenced plugins and their runtime dependency-subgraphs, which you'll require in a cleanroom environment in order to be able to execute the build.

    The full listing of current presets include:

    • none - This uses an AnyFilter, which means no filter at all. It traverses ALL relationships, and is quite exhaustive, as if you wanted to rebuild your projects and everything in their build-time environment. Recursively. You probably don't mean to use this filter, except maybe as a curiosity.
    • sob - Explained above.
    • sob-build - This roughly equates to the Maven runtime dependency closure, returning everything you might want to rebuild for your project.

  4. Query the resolved dependency graph.

    The simplest way to do this might be to generate a modified dependency:tree output for the graph:

    ```bash
$ curl 'http://localhost:8080/aprox/api/1.0/depgraph/render/tree/org.commonjava.aprox.wars/aprox-savant/0.8.1?wsid=409&preset=sob'

org.commonjava.aprox.wars:aprox-savant:0.8.1
  org.commonjava.aprox.wars:aprox-wars:0.8.1:pom:PARENT
    org.commonjava.aprox:aprox-parent:0.8.1:pom:PARENT
  org.commonjava.aprox:aprox-api:0.8.1:jar:compile (0)
    org.commonjava.aprox:aprox-parent:0.8.1:pom:PARENT
    org.commonjava.util:configuration-api:0.4:jar:compile (0)
    com.google.code.gson:gson:2.2.2:jar:compile (1)
    org.commonjava.web:json-serialization:0.5:jar:compile (2)
    commons-lang:commons-lang:2.6:jar:compile (3)
      org.apache.commons:commons-parent:17:pom:PARENT
        org.apache:apache:7:pom:PARENT
          org.apache:apache:7:pom:PARENT
    org.commonjava.maven.galley:galley-transport-httpclient:0.1.0:jar:compile (5)
    org.commonjava.maven.galley:galley-core:0.1.0:jar:compile (6)
  org.commonjava.aprox:aprox-core:0.8.1:jar:compile (1)
    org.commonjava.aprox:aprox-parent:0.8.1:pom:PARENT
    org.apache.maven.archetype:archetype-catalog:2.2:jar:compile (0)
    org.commonjava.util:configuration-dotconf:0.4:jar:compile (7)
    org.apache.maven:maven-repository-metadata:3.0.3:jar:compile (8)
    org.hamcrest:hamcrest-core:1.1:jar:test (10)
    org.jboss.weld.se:weld-se-core:1.1.8.Final:jar:test (11)
    org.commonjava.qarqas:qarqas-test-harness:0.3:jar:test (12)
...
```

    You'll notice the presence of PARENT listings. This is why I call it modified dependency:tree. The original plugin doesn't include these, which makes it hard to know where they're used, or what list of parent POMs need to be included when constructing a repository. For dependencies listed in this output, you'll notice a number at the end; this is the index of the dependency in the relevant POM file. If there are gaps in the sequence for any given POM, it's because this tree listing is suppressing duplicates in the same way Maven's collision management logic works.

  5. (OPTIONAL) Delete the workspace when you're done with it:

    $ curl -X DELETE http://localhost:8080/aprox/api/1.0/depgraph/ws/409

    I say this part is optional because the real strength of workspaces it their durability. You can setup and fine-tune a workspace for use with many related operations...say, for working on many projects in a single new environment you're setting up.

Special Workflows

Certain of Depgraph's features don't follow the above workflow. This is usually because they require more advanced configuration, or may try to consolidate multiple steps for convenience, or both.

NOTE: In all cases, these operations will still perform better if you create a workspace and configure it as needed. See above, in General Workflow for basic instructions.

Download Manifests

Download manifests are various ways of outputting a list of downloadable URLs for artifacts in a particular (filtered?) depgraph. These are particularly useful for handing off to external tools for the purpose of bootstrapping a repository specially for a project build, or something similar. Currently, these include:

  • urlmap (URL path: /aprox/api/1.0/depgraph/downloads/urlmap) - This is a JSON mapping of GAV to a list of URLs that correspond to the main artifact, POM, and any attachments that are enabled by configuration.
  • downlog (URL path: /aprox/api/1.0/depgraph/downloads/downlog) - This is a simple list of downloadable URLs, prefixed by 'Downloading ' to simulate a much denser version of the console output from an actual Maven build. The URLs will include main artifact, POM, and any attachments that are enabled by configuration.

All download-manifest operations include a resolve step that will discover the dependency graph before generating the output. This saves the need to resolve then query in two steps. It doesn't incur a lot of extra penalty in URL complexity because these operations already require the use of a POST'ed JSON configuration to function properly.

The basic configuration looks like this:

{
  "roots": [
    "org.commonjava.aprox.wars:aprox-savant:0.8.1"
  ],
  "extras": [
    {
      "classifier": "sources",
      "type": "jar",
    },
    {
      "classifier": "javadoc"
    }
  ],
  "workspaceId": "409",
  "source": "group:public",
  "preset": "sob-build",
  "resolve": true
}

As you can see, this configuration is designed to allow multiple roots and multiple extra, attached artifacts in the generated output. Multiple roots are useful if you're building more than one top-level project in your environment, and these top-level projects may have overlapping depgraphs. Multiple attached artifacts are useful when you want to certain extras beyond the simple runtime bits in your repository. For instance, if you want to generate a repository to support IDEs, which often rely heavily on source jars.

NOTE: I've included a type field in one of the extras sections. The type jar is the default; I only included it here to illustrate that as a configuration point.

The urlmap Command

The basic command looks like this:

curl -X POST -H 'Content-Type: application/json' --data @config.json http://localhost:8080/aprox/api/1.0/depgraph/downloads/urlmap
{
  "org.commonjava.maven.atlas:atlas-parent:0.8.0": [
    "http://localhost:8080/aprox/api/1.0/repository/central/org/commonjava/maven/atlas/atlas-parent/0.8.0/atlas-parent-0.8.0.pom"
  ],
  "org.neo4j:neo4j-udc:1.9.RC2": [
    "http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-udc/1.9.RC2/neo4j-udc-1.9.RC2-sources.jar",    
    "http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-udc/1.9.RC2/neo4j-udc-1.9.RC2.jar", 
    "http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-udc/1.9.RC2/neo4j-udc-1.9.RC2-javadoc.jar", 
    "http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-udc/1.9.RC2/neo4j-udc-1.9.RC2.pom"
  ],
  "commons-codec:commons-codec:1.6": [
    "http://localhost:8080/aprox/api/1.0/repository/central/commons-codec/commons-codec/1.6/commons-codec-1.6.jar", 
    "http://localhost:8080/aprox/api/1.0/repository/central/commons-codec/commons-codec/1.6/commons-codec-1.6.pom", 
    "http://localhost:8080/aprox/api/1.0/repository/central/commons-codec/commons-codec/1.6/commons-codec-1.6-sources.jar", 
    "http://localhost:8080/aprox/api/1.0/repository/central/commons-codec/commons-codec/1.6/commons-codec-1.6-javadoc.jar"
  ],
  "org.neo4j:neo4j-kernel:1.9.RC2": [
    "http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-kernel/1.9.RC2/neo4j-kernel-1.9.RC2-javadoc.jar", 
    "http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-kernel/1.9.RC2/neo4j-kernel-1.9.RC2.jar", 
    "http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-kernel/1.9.RC2/neo4j-kernel-1.9.RC2.pom", 
    "http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-kernel/1.9.RC2/neo4j-kernel-1.9.RC2-sources.jar"
  ],
...
}

The downlog Command

The basic command looks like this:

curl -X POST -H 'Content-Type: application/json' --data @config.json http://localhost:8080/aprox/api/1.0/depgraph/downloads/downlog
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j/1.9.RC2/neo4j-1.9.RC2-sources.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/apache/lucene/lucene-core/3.6.2/lucene-core-3.6.2.pom
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/commonjava/maven/atlas/atlas-identities/0.8.0/atlas-identities-0.8.0-javadoc.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/apache/lucene/lucene-parent/3.6.2/lucene-parent-3.6.2.pom
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/commonjava/commonjava/2/commonjava-2.pom
Downloading http://localhost:8080/aprox/api/1.0/repository/central/com/googlecode/concurrentlinkedhashmap/concurrentlinkedhashmap-lru/1.3.1/concurrentlinkedhashmap-lru-1.3.1-javadoc.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-graph-matching/1.9.RC2/neo4j-graph-matching-1.9.RC2-javadoc.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-graph-algo/1.9.RC2/neo4j-graph-algo-1.9.RC2-javadoc.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/slf4j/slf4j-parent/1.6.1/slf4j-parent-1.6.1.pom
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/commonjava/maven/atlas/atlas-drivers-parent/0.8.0/atlas-drivers-parent-0.8.0.pom
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/slf4j/slf4j-api/1.6.1/slf4j-api-1.6.1-sources.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/commons-lang/commons-lang/2.6/commons-lang-2.6.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/apache/lucene/lucene-core/3.6.2/lucene-core-3.6.2.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-graph-matching/1.9.RC2/neo4j-graph-matching-1.9.RC2.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/commonjava/maven/atlas/atlas-relationships-api/0.8.0/atlas-relationships-api-0.8.0-sources.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-udc/1.9.RC2/neo4j-udc-1.9.RC2.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/apache/commons/commons-parent/17/commons-parent-17.pom
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/commonjava/maven/atlas/atlas-identities/0.8.0/atlas-identities-0.8.0.pom
Downloading http://localhost:8080/aprox/api/1.0/repository/central/commons-codec/commons-codec/1.6/commons-codec-1.6-sources.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-udc/1.9.RC2/neo4j-udc-1.9.RC2-sources.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/commonjava/maven/atlas/atlas-parent/0.8.0/atlas-parent-0.8.0.pom
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/apache/lucene/lucene-core/3.6.2/lucene-core-3.6.2-sources.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/log4j/log4j/1.2.12/log4j-1.2.12.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-jmx/1.9.RC2/neo4j-jmx-1.9.RC2-sources.jar
Downloading http://localhost:8080/aprox/api/1.0/repository/central/org/neo4j/neo4j-kernel/1.9.RC2/neo4j-kernel-1.9.RC2-sources.jar
...
Add a custom footer
Add a custom sidebar
Clone this wiki locally