Bug traversing GremlinPipeline with out() and outE().inV()

[pellyadolfo]

As a follow up of this thread in groups (https://groups.google.com/forum/#!topic/orient-database/rGMofFJ-Pww), as I my responses are automatically deleted, I post a new issue here.

The following code includes a test case for this:

    OGraphDatabase database = OGraphDatabasePool.global().acquire("remote:localhost/onto", "admin", "admin");
    OSchema schema = database.getMetadata().getSchema();
    ODocument sourceDoc = null;
    ODocument targetDoc = null;
    ODocument edge = null; 
    try {
        schema.createClass("SourceNodeType", database.getMetadata().getSchema().getClass("V"));
        schema.createClass("TargetNodeType", database.getMetadata().getSchema().getClass("V"));

        sourceDoc = database.createVertex("SourceNodeType").save();
        targetDoc = database.createVertex("TargetNodeType").save();
        edge = database.createEdge(sourceDoc, targetDoc).field("label", "app").save();
        database.commit();

        final GremlinPipeline pipeline = new GremlinPipeline()
        .start(new OrientGraph("remote:localhost/onto", "admin", "admin").getVerticesOfClass("SourceNodeType"))
        .filter(new PipeFunction<Vertex, Boolean>() {
            public Boolean compute(Vertex vertex) {
                System.out.println("1vertex->" + vertex + "; props: " + vertex.getPropertyKeys());
                return true;
            }
        })
        //.out()
        //.outE().inV()
        .filter(new PipeFunction<Vertex, Boolean>() {
            public Boolean compute(Vertex vertex) {
                System.out.println("2vertex->" + vertex + "; props: " + vertex.getPropertyKeys());
                return true;
            }
        });     

        System.out.println(pipeline.iterator().next());


    } finally {
        database.removeEdge(edge);
        database.removeVertex(sourceDoc);
        database.removeVertex(targetDoc);
        schema.dropClass("SourceNodeType");
        schema.dropClass("TargetNodeType");
        database.close();
    }

Code is adding a simple structure

source ---------------------> target

When lines in the middle of both filters are commented, result is as expected:

1vertex->v(SourceNodeType)[#26:0]; props: [out]
2vertex->v(SourceNodeType)[#26:0]; props: [out]
v(SourceNodeType)[#26:0]

When any of the lines in the middle of the filters is uncommented, result is:

1vertex->v(SourceNodeType)[#26:0]; props: [out]
Exception in thread "main" java.util.NoSuchElementException
at com.tinkerpop.blueprints.impls.orient.OrientElementIterator.next(OrientElementIterator.java:32)
at com.tinkerpop.blueprints.impls.orient.OrientElementIterator.next(OrientElementIterator.java:13)
at com.tinkerpop.pipes.util.iterators.HistoryIterator.next(HistoryIterator.java:25)
at com.tinkerpop.pipes.IdentityPipe.processNextStart(IdentityPipe.java:19)
at com.tinkerpop.pipes.AbstractPipe.next(AbstractPipe.java:89)
at com.tinkerpop.pipes.filter.FilterFunctionPipe.processNextStart(FilterFunctionPipe.java:23)
at com.tinkerpop.pipes.AbstractPipe.next(AbstractPipe.java:89)
at com.tinkerpop.pipes.transform.VertexQueryPipe.processNextStart(VertexQueryPipe.java:85)
at com.tinkerpop.pipes.transform.VertexQueryPipe.processNextStart(VertexQueryPipe.java:19)
at com.tinkerpop.pipes.AbstractPipe.next(AbstractPipe.java:89)
at com.tinkerpop.pipes.transform.EdgesVerticesPipe.processNextStart(EdgesVerticesPipe.java:37)
at com.tinkerpop.pipes.transform.EdgesVerticesPipe.processNextStart(EdgesVerticesPipe.java:12)
at com.tinkerpop.pipes.AbstractPipe.next(AbstractPipe.java:89)
at com.tinkerpop.pipes.filter.FilterFunctionPipe.processNextStart(FilterFunctionPipe.java:23)
at com.tinkerpop.pipes.AbstractPipe.next(AbstractPipe.java:89)
at com.tinkerpop.pipes.util.Pipeline.next(Pipeline.java:115)
at com.vaalbara.dao.Test2.main(Test2.java:59)

Result of queries:

• select from W3C Web SQL Database compliant? [moved] #26:0 ---------------> [SourceNodeType#26:0{out:[1]} v1]
• select flatten(out.in) from W3C Web SQL Database compliant? [moved] #26:0 ---------------------> [a suggestion of kv server 's memory model [moved] #27:0]
  so results are there.

It looks a bug in traversing the GremlinPipeline with out() and outE().inV().

[pellyadolfo]

A second bug for the same test case in this query:

List<Vertex> sources = new GremlinPipeline()
.start(new OrientGraph("remote:localhost/onto", "admin", "admin").getVerticesOfClass("SourceNodeType")).
.outE().inV()
.toList();
System.out.println(sources);

In this second case:

• start() fetches the vertex SourceNodeType,
• outE().inV() should navigate to the vertex TargetNodeSource

so I would expected System.out be printing [the_target_node] but result is [] instead.

Same thing if using .out() to traverse

[lvca]

Please may you provide me a database with such vertices to debug what happens?

[pellyadolfo]

The piece of code above creates and drops the 2 vertices and edge required to recreate the issue.

sourceDoc = database.createVertex("SourceNodeType").save();
targetDoc = database.createVertex("TargetNodeType").save();
edge = database.createEdge(sourceDoc, targetDoc).field("label", "app").save();

Just copy and paste the code in first message on any database and you will be able to recreate the issue.

There is a second issue in the second message as well

[pellyadolfo]

I think I have narrowed down the issue by trying to use a generic transform() pipe (where out(), outE() and the like comes from).

If we place the following transform pipe instead out() in the test case:

        .transform(new PipeFunction<Vertex, Vertex>() {
            public Vertex compute(Vertex vertex) {
                try {
                    System.out.println("transform->" + vertex.getEdges(Direction.OUT).iterator().next());
                } catch (Exception e) {
                    e.printStackTrace();
                }
                return vertex;
            }
        })

the same exception is thrown once again in vertex.getEdges(Direction.OUT).iterator().next().

Therefore it looks a problem fetching edges of a vertex by using Blueprints API. It would explain all exceptions reported in this thread.

java.util.NoSuchElementException
at com.orientechnologies.common.collection.OMultiCollectionIterator.next(OMultiCollectionIterator.java:85)
at com.vaalbara.dao.Test2$2.compute(Test2.java:52)
at com.vaalbara.dao.Test2$2.compute(Test2.java:1)
at com.tinkerpop.pipes.transform.TransformFunctionPipe.processNextStart(TransformFunctionPipe.java:22)
at com.tinkerpop.pipes.AbstractPipe.next(AbstractPipe.java:89)
at com.tinkerpop.pipes.filter.FilterFunctionPipe.processNextStart(FilterFunctionPipe.java:23)
at com.tinkerpop.pipes.AbstractPipe.next(AbstractPipe.java:89)
at com.tinkerpop.pipes.util.Pipeline.next(Pipeline.java:115)
at com.tinkerpop.pipes.util.PipeHelper.fillCollection(PipeHelper.java:52)
at com.tinkerpop.gremlin.java.GremlinPipeline.toList(GremlinPipeline.java:1564)
at com.vaalbara.dao.Test2.main(Test2.java:65)

By the way, same issue for vertex.getVertices(Direction.OUT).iterator().next().

[pellyadolfo]

found this message in list

https://groups.google.com/forum/#!topic/orient-database/sXOVuCF5Om0

stating that OrientVertex.getEdges() was fixed for latest snapshots

Tested the test case in this thread with very latest snapshot orientdb-1.6.0-20131011.195757-105-distribution and still same exceptions are thrown.

[lvca]

Adolfo, if you can provide me a test case for this I'm pretty confident I can fix in short time.

[pellyadolfo]

Luca, the test case is in the first message of this thread.

[lvca]

I see you connected to a remote db, so I guess I'd need it. However I replaced remote:... to memory: and everything works. I've also removed #26:0 by using the sourceDoc's identity:

public class Snippet {
  @Test
  public void test() {
    OGraphDatabase database = new OGraphDatabase("memory:onto").create();
...
      List<ODocument> result = database.query(new OSQLSynchQuery<ODocument>("select from " + sourceDoc.getIdentity()));
      System.out.println(result);
      result = database.query(new OSQLSynchQuery<ODocument>("select flatten(out.in) from " + sourceDoc.getIdentity()));
...
  }
}

So is it a problem of remote protocol?

[pellyadolfo]

Can you please try to add the remote storage and invoke remotely? (just run test case)

<storage name="onto" path="local:C:\orientdb-1.6.0-SNAPSHOT\databases\onto" userName="admin" userPassword="admin" loaded-at-startup="true"/>

yes, I am creating remotely 2 vertices and an edge and querying remotely as well.

It looks more an issue with next() of the iterator. Maybe running next() remotely is accessing disk and not when done in memory. Not idea.

(But, it does not need to be right now....)

[willmoss]

Hi -

I also have encountered this bug.

In gremlin, start().outE().inV() returns [ ]

[lvca]

The problem is how you create the graph using the OLD GraphDB API. Try this and works:

public class Snippet {
  @Test
  public void test() {
    OrientGraph graph = new OrientGraph("remote:localhost/tinkerpop", "admin", "admin");
    Vertex sourceDoc = null;
    Vertex targetDoc = null;
    Edge edge = null;
    try {
      graph.createVertexType("SourceNodeType");
      graph.createVertexType("TargetNodeType");

      sourceDoc = graph.addVertex("class:SourceNodeType");
      targetDoc = graph.addVertex("class:TargetNodeType");
      edge = sourceDoc.addEdge("app", targetDoc);
      graph.commit();

      final GremlinPipeline pipeline = new GremlinPipeline()
          .start(new OrientGraph("remote:localhost/tinkerpop", "admin", "admin").getVerticesOfClass("SourceNodeType"))
          .filter(new PipeFunction<Vertex, Boolean>() {
            public Boolean compute(Vertex vertex) {
              System.out.println("1vertex->" + vertex + "; props: " + vertex.getPropertyKeys());
              return true;
            }
          }).out()
          // .outE().inV()
          .filter(new PipeFunction<Vertex, Boolean>() {
            public Boolean compute(Vertex vertex) {
              System.out.println("2vertex->" + vertex + "; props: " + vertex.getPropertyKeys() + "; doc: "
                  + ((OrientVertex) vertex).getRecord());
              return true;
            }
          });

      System.out.println(pipeline.iterator().next());

    } finally {

      graph.removeEdge(edge);
      graph.removeVertex(sourceDoc);
      graph.removeVertex(targetDoc);
      graph.dropVertexType("SourceNodeType");
      graph.dropVertexType("TargetNodeType");
      graph.shutdown();
    }
  }
}

[lvca]

The problem is that old native API uses "out" and "in" for vertices edges. The new one has "out_" and "in_", so I've added a new OGraphDatabase.setRetroCompatibility(true) that allows to open database created with previous releases. But now by default uses the new one that is compatible with TinkerPop stack and Gremlin of course.

To convert old database with the new format we've a migration tool:

java -cp target/classes/com.orientechnologies.orient.core.db.graph.OGraphDatabaseMigration <database-url> [<user> <password>]

It converts all "out" -> "out_" and "in" -> "in_".

[luigi-lt]

Could you attach an example of the migration tool command (in particular database-url section) ? I have this "in"/"out" bug, too. I'm using an embedded Orient database and sometimes I don't receive all the expected vertices.

[lvca]

Example:

java -cp $ORIENTDB_HOME/lib/*.jar target/classes/com.orientechnologies.orient.core.db.graph.OGraphDatabaseMigration local:/temp/mydb admin admin

[luigi-lt]

Can I substitute $ORIENTDB_HOME with the path to the OrientDB's jars in my Eclipse project? (please remember that I'm embedding OrientDB in my application)

[lvca]

The important is giving the path of libraries