Skip to content

Commit

Permalink
feat: Add support for deleteWhere in ObjectBoxVectorStore (#552)
Browse files Browse the repository at this point in the history
  • Loading branch information
davidmigloz authored Sep 25, 2024
1 parent 1da543f commit 90918bb
Show file tree
Hide file tree
Showing 5 changed files with 221 additions and 9 deletions.
87 changes: 85 additions & 2 deletions docs/modules/retrieval/vector_stores/integrations/objectbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,14 @@ To delete documents, you can use the `delete` method passing the ids of the docu
await vectorStore.delete(ids: ['9999']);
```

You can also use `deleteWhere` to delete documents based on a condition.

```dart
await vectorStore.deleteWhere(
ObjectBoxDocumentProps.metadata.contains('cat'),
);
```

## Example: Building a Fully Local RAG App with ObjectBox and Ollama

This example demonstrates how to build a fully local RAG (Retrieval-Augmented Generation) app using ObjectBox and Ollama. The app retrieves blog posts, splits them into chunks, and stores them in an ObjectBox vector store. It then uses the stored information to generate responses to user questions.
Expand Down Expand Up @@ -250,7 +258,7 @@ Check out the [Wikivoyage EU example](https://github.com/davidmigloz/langchain_d

### BaseObjectBoxVectorStore

If you need more control over the entity (e.g. if you need to persist custom fields), you can use the `BaseObjectBoxVectorStore` class instead of `ObjectBoxVectorStore`.
If you need more control over the entity (e.g. if you are using ObjectBox to store other entities, or if you need to customize the Document entity class.), you can use the `BaseObjectBoxVectorStore` class instead of `ObjectBoxVectorStore`.

`BaseObjectBoxVectorStore` requires the following parameters:
- `embeddings`: The embeddings model to use.
Expand All @@ -260,4 +268,79 @@ If you need more control over the entity (e.g. if you need to persist custom fie
- `getIdProperty`: A function that returns the ID property of the entity.
- `getEmbeddingProperty`: A function that returns the embedding property of the entity.

You can check how `ObjectBoxVectorStore` is implemented to see how to use `BaseObjectBoxVectorStore`.
Here is an example of how to use this class:

First, you can define our own Document entity class instead of using the one provided by the [ObjectBoxVectorStore]. In this way, you can customize the entity to your needs. You will need to define the mapping logic between the entity and the LangChain [Document] model.

```dart
@Entity()
class MyDocumentEntity {
MyDocumentEntity({
required this.id,
required this.content,
required this.metadata,
required this.embedding,
});
@Id()
int internalId = 0;
@Unique(onConflict: ConflictStrategy.replace)
String id;
String content;
String metadata;
@HnswIndex(
dimensions: 768,
distanceType: VectorDistanceType.cosine,
)
@Property(type: PropertyType.floatVector)
List<double> embedding;
factory MyDocumentEntity.fromModel(
Document doc, List<double> embedding,
) => MyDocumentEntity(
id: doc.id ?? '',
content: doc.pageContent,
metadata: jsonEncode(doc.metadata),
embedding: embedding,
);
Document toModel() => Document(
id: id,
pageContent: content,
metadata: jsonDecode(metadata),
);
}
```

After defining the entity class, you will need to run the ObjectBox generator:

```sh
dart run build_runner build --delete-conflicting-outputs
```

Then, you just need to create your custom vector store class that extends [BaseObjectBoxVectorStore] and wire everything up:

```dart
class MyCustomVectorStore extends BaseObjectBoxVectorStore<MyDocumentEntity> {
MyCustomVectorStore({
required super.embeddings,
required Store store,
}) : super(
box: store.box<MyDocumentEntity>(),
createEntity: (
String id,
String content,
String metadata,
List<double> embedding,
) =>
MyDocumentEntity(
id: id,
content: content,
metadata: metadata,
embedding: embedding,
),
createDocument: (MyDocumentEntity docDto) => docDto.toModel(),
getIdProperty: () => MyDocumentEntity_.id,
getEmbeddingProperty: () => MyDocumentEntity_.embedding,
);
}
```

Now you can use the [MyCustomVectorStore] class to store and search documents.
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,101 @@ import 'package:uuid/uuid.dart';
/// {@template base_object_box_vector_store}
/// Base class for ObjectBox vector store.
///
/// Use this class if you need more control over the ObjectBox store.
/// Otherwise, use [ObjectBoxVectorStore] which is a pre-configured version.
/// The [ObjectBoxVectorStore] class is a pre-configured version of this class,
/// but it can only be used if you don't use ObjectBox for anything else.
///
/// If you need more control over the ObjectBox store, use this class instead.
/// For example, if you are using ObjectBox to store other entities, or if you
/// need to customize the Document entity class.
///
/// Here is an example of how to use this class:
///
/// First, you can define our own Document entity class instead of using the
/// one provided by the [ObjectBoxVectorStore]. In this way, you can customize
/// the entity to your needs. You will need to define the mapping logic between
/// the entity and the LangChain [Document] model.
///
/// ```dart
/// @Entity()
/// class MyDocumentEntity {
/// MyDocumentEntity({
/// required this.id,
/// required this.content,
/// required this.metadata,
/// required this.embedding,
/// });
///
/// @Id()
/// int internalId = 0;
///
/// @Unique(onConflict: ConflictStrategy.replace)
/// String id;
///
/// String content;
///
/// String metadata;
///
/// @HnswIndex(
/// dimensions: 768,
/// distanceType: VectorDistanceType.cosine,
/// )
/// @Property(type: PropertyType.floatVector)
/// List<double> embedding;
///
/// factory MyDocumentEntity.fromModel(
/// Document doc, List<double> embedding,
/// ) => MyDocumentEntity(
/// id: doc.id ?? '',
/// content: doc.pageContent,
/// metadata: jsonEncode(doc.metadata),
/// embedding: embedding,
/// );
///
/// Document toModel() => Document(
/// id: id,
/// pageContent: content,
/// metadata: jsonDecode(metadata),
/// );
/// }
/// ```
///
/// After defining the entity class, you will need to run the ObjectBox
/// generator:
///
/// ```sh
/// dart run build_runner build --delete-conflicting-outputs
/// ```
///
/// Then, you just need to create your custom vector store class that
/// extends [BaseObjectBoxVectorStore] and wire everything up:
///
/// ```dart
/// class MyCustomVectorStore extends BaseObjectBoxVectorStore<MyDocumentEntity> {
/// MyCustomVectorStore({
/// required super.embeddings,
/// required Store store,
/// }) : super(
/// box: store.box<MyDocumentEntity>(),
/// createEntity: (
/// String id,
/// String content,
/// String metadata,
/// List<double> embedding,
/// ) =>
/// MyDocumentEntity(
/// id: id,
/// content: content,
/// metadata: metadata,
/// embedding: embedding,
/// ),
/// createDocument: (MyDocumentEntity docDto) => docDto.toModel(),
/// getIdProperty: () => MyDocumentEntity_.id,
/// getEmbeddingProperty: () => MyDocumentEntity_.embedding,
/// );
/// }
/// ```
///
/// Now you can use the [MyCustomVectorStore] class to store and search documents.
/// {@endtemplate}
class BaseObjectBoxVectorStore<T> extends VectorStore {
/// {@macro base_object_box_vector_store}
Expand Down Expand Up @@ -87,8 +180,15 @@ class BaseObjectBoxVectorStore<T> extends VectorStore {
}

@override
Future<void> delete({required final List<String> ids}) async {
_box.query(_getIdProperty().oneOf(ids)).build().remove();
Future<void> delete({required final List<String> ids}) {
return _box.query(_getIdProperty().oneOf(ids)).build().removeAsync();
}

/// Delete by condition.
///
/// - [condition] is the condition to delete by.
Future<void> deleteWhere(final Condition<T> condition) {
return _box.query(condition).build().removeAsync();
}

@override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ import 'types.dart';
///
/// This vector stores creates a [Store] with an [ObjectBoxDocument] entity
/// that persists LangChain [Document]s along with their embeddings. If you
/// need more control over the entity, you can use the
/// need more control over the entity or the storeo, you can use the
/// [BaseObjectBoxVectorStore] class instead.
///
/// See documentation for more details:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,37 @@ void main() async {
);
expect(res2.length, 0);
});

test('Test delete where', () async {
await vectorStore.addDocuments(
documents: [
const Document(
id: '9999',
pageContent: 'This document will be deleted',
metadata: {'cat': 'xxx'},
),
],
);
final res1 = await vectorStore.similaritySearch(
query: 'Deleted doc',
config: ObjectBoxSimilaritySearch(
filterCondition: ObjectBoxDocumentProps.metadata.contains('xxx'),
),
);
expect(res1.length, 1);
expect(res1.first.id, '9999');

await vectorStore.deleteWhere(
ObjectBoxDocumentProps.metadata.contains('xxx'),
);
final res2 = await vectorStore.similaritySearch(
query: 'Deleted doc',
config: ObjectBoxSimilaritySearch(
filterCondition: ObjectBoxDocumentProps.metadata.contains('xxx'),
),
);
expect(res2.length, 0);
});
});

group('ObjectBoxSimilaritySearch', () {
Expand Down
2 changes: 0 additions & 2 deletions packages/langchain_core/lib/src/vector_stores/base.dart
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,6 @@ abstract class VectorStore {
/// Delete by vector ID.
///
/// - [ids] is a list of ids to delete.
///
/// Returns true if the delete was successful.
Future<void> delete({required final List<String> ids});

/// Returns docs most similar to query using specified search type.
Expand Down

0 comments on commit 90918bb

Please sign in to comment.