Binary Data

Consideration about the management of binary data

Introduction

OrientDB natively handles binary data, namely BLOB. However there are some consideration to do based on the type of binary data, the size, the kind of usage, etc.

Sometimes it's better to store binary records in a different path then default database directory to benefit of faster HD (like a SSD) or just to go in parallel if the OS and HW configuration allow this.

In this case create a new data-segment and cluster in a different path (available since v1.0):

    db.addDataSegment("binary", "/mnt/ssd");
    db.addCluster("physical", "binary", "/mnt/ssd", "binary" );

All the records in cluster "binary" will reside in files created under the directory "/mnt/ssd".

Techniques

Store on file system and save the path in the document

This is the simpler way to handle binary data: store them to the file system and just keep the path to retrieve them.

Example:

    ODocument doc = new ODocument(db);
    doc.field("binary", "/usr/local/orientdb/binary/test.pdf");
    doc.save();

Pros:

• Easy to write
• 100% delegated to the File System

Cons:

• Binary data can't be automatically distributed using the OrientDB cluster

Store it as Document's field

ODocument class is able to manage binary data in form of {{{byte[(byte array). Example:

    ODocument doc = new ODocument(db);
    doc.field("binary", "Binary data".getBytes());
    doc.save();

This is the easiest way to keep the binary data inside the database, but it's not really efficient on large BLOB because the binary content is serialized in Base64. This means a waste of space (33% more) and a run-time cost in marshalling/unmarshalling.

Pros:

• Easy to write

Cons:

• Waste of space +33%
• Run-time cost of marshalling/unmarshalling

Store it with ORecordBytes

The ORecordBytes class is a record implementation able to store binary content without conversions (see above). This is the faster way to handle binary data with OrientDB but needs a separate record to handle it.

Example:

    ORecordBytes record = new ORecordBytes(db, "Binary data".getBytes());
    record.save();

Since this is a separate record, the best way to reference it is to link it to a Document record. Example:

ORecordBytes record = new ORecordBytes(db, "Binary data".getBytes());

ODocument doc = new ODocument(db);
doc.field("id", 12345);
doc.field("binary", record);
doc.save();

In this way you can access to the binary data by traversing the "binary" field of the parent's document record.

ORecordBytes record = doc.field("binary");
byte[] content = record.toStream();

ORecordBytes class can work with Java Streams:

ORecordBytes record = new ORecordBytes(db).fromInputStream(in);

record.toOutputStream(out);

Pros:

• Fast and compact solution

Cons:

• Slightly complex management

Large content: split in multiple ORecordBytes

OrientDB can store up to 2Gb as record content. But there are other limitations on network buffers and file sizes you should tune to reach the 2GB barrier.

However managing big chunks of binary data means having big byte[] structures in RAM and this could cause a Out Of Memory of the JVM. Many users reported that splitting the binary data in chunks it's the best solution.

Continuing from the last example we could handle not a single reference against one ORecordBytes record but multiple references. A One-To-Many relationship. For this purpose the LINKLIST type fits perfect because maintains the order.

To avoid OrientDB caches in memory large records use the massive insert intent and keep in the collection the RID, not the entire records.

Example to store in OrientDB the file content:

    database.declareIntent( new OIntentMassiveInsert() );
    
    List<ORID> chunks = new ArrayList<ORID>();
    InputStream in = new BufferedInputStream( new FileInputStream( file ) );
    while ( in.available() > 0 ) {
      final ORecordBytes chunk = new ORecordBytes();
    
      // READ REMAINING DATA, BUT NOT MORE THAN 8K
      chunk.fromInputStream( in, 8192 );
    
      // SAVE THE CHUNK TO GET THE REFERENCE (IDENTITY) AND FREE FROM THE MEMORY
      database.save( chunk );
    
      // SAVE ITS REFERENCE INTO THE COLLECTION
      chunks.add( chunk.getIdentity() );
    }
    
    // SAVE THE COLLECTION OF REFERENCES IN A NEW DOCUMENT
    ODocument record = new ODocument();
    record.field( "data", chunks );
    database.save( record );
    
    database.declareIntent( null );

Example to read back the file content:

    record.setLazyLoad(false);
    for (OIdentifiable id : (List<OIdentifiable>) record.field("data")) {
      ORecordBytes chunk = (ORecordBytes) id.getRecord();
      chunk.toOutputStream(out);
      chunk.unload();
    }

Pros:

• Fastest and compact solution

Cons:

• More complex management

Conclusion

What to use?

• Have you short binary data? Store them as document's field
• Do you want the maximum of performance and better use of the space? Store it with ORecordBytes
• Have you large binary objects? Store it with ORecordBytes but split the content in multiple records