generated from wesen/wesen-go-template
-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #33 from wesen/task/add-embeddings
Add embeddings support and profile settings layer
- Loading branch information
Showing
18 changed files
with
569 additions
and
42 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,67 @@ | ||
| ## Enhanced Error Handling with Raw Results | ||
|
|
||
| Added support for printing raw error responses when the --raw-results flag is enabled. This helps with debugging by showing the complete error response from Elasticsearch. | ||
|
|
||
| - Print complete error response to stderr when raw-results is enabled | ||
| - Print error reason and root cause to stderr when raw-results is disabled | ||
|
|
||
| # Refactor Embeddings Settings Factory | ||
|
|
||
| Simplified the embeddings settings factory to use a minimal configuration struct instead of depending on the full StepSettings. Added backwards compatibility method. | ||
|
|
||
| - Created new EmbeddingsConfig struct for minimal configuration | ||
| - Modified SettingsFactory to use EmbeddingsConfig instead of StepSettings | ||
| - Added NewSettingsFactoryFromStepSettings for backwards compatibility | ||
|
|
||
| # Fix Embeddings Settings Type Handling | ||
|
|
||
| Fixed type handling in embeddings settings to properly handle pointer types in StepSettings and non-pointer types in EmbeddingsConfig. | ||
|
|
||
| - Updated CreateEmbeddingsConfig to properly dereference pointer types | ||
| - Modified NewProvider to handle non-pointer types in EmbeddingsConfig | ||
| - Fixed error checks to use empty string checks instead of nil checks | ||
|
|
||
| # Add Provider Options for Embeddings Factory | ||
|
|
||
| Added functional options pattern to the embeddings provider factory for more flexible configuration. | ||
|
|
||
| - Added WithType, WithEngine, WithBaseURL, WithAPIKey, and WithDimensions option functions | ||
| - Modified NewProvider to accept variadic options | ||
| - Improved configuration handling with options overriding defaults | ||
|
|
||
| # Add Custom Tags Documentation | ||
|
|
||
| Added comprehensive documentation for implementing custom tags in go-emrichen, including: | ||
| - Basic tag implementation patterns | ||
| - Argument handling and validation | ||
| - Environment interaction | ||
| - Node processing utilities | ||
| - Testing guidelines and best practices | ||
| - Conceptual explanations and rationale for design patterns | ||
| - Detailed best practices and common patterns | ||
| - In-depth discussion of error handling and type safety | ||
|
|
||
| # Fix Custom Tags Documentation Signature | ||
|
|
||
| Updated custom tags documentation to reflect correct function signature: | ||
| - Changed tag handler signature to include interpreter parameter | ||
| - Clarified pure function nature of tag handlers | ||
| - Updated all code examples to use correct signature | ||
| - Added explanation of interpreter parameter usage | ||
|
|
||
| # Enhance Custom Tags Documentation with ParseArgs Guidelines | ||
|
|
||
| Added detailed documentation about argument handling and recursive processing: | ||
| - Comprehensive guide to using ParseArgs | ||
| - Core principles for implementing tag handlers | ||
| - Examples of proper argument validation and processing | ||
| - Guidelines for recursive processing of nested structures | ||
| - Detailed error handling patterns for arguments | ||
|
|
||
| # Update Custom Tags Documentation with Proper Namespace | ||
|
|
||
| Updated custom tags documentation to use proper import paths and namespaces: | ||
| - Added proper import statements for all examples | ||
| - Updated all type references to use emrichen namespace | ||
| - Fixed function signatures to use emrichen.Interpreter | ||
| - Updated utility function calls to use emrichen namespace |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,36 @@ | ||
| name: "search-insights" | ||
| short: "Search through summaries using embeddings and keywords" | ||
| long: "Search through the summaries using both semantic similarity via embeddings and keyword/fuzzy matching for better results" | ||
|
|
||
| flags: | ||
| - name: query | ||
| type: string | ||
| help: "Search text to find similar content" | ||
| required: true | ||
| - name: k | ||
| type: int | ||
| help: "Number of results to return" | ||
| default: 5 | ||
|
|
||
| default-index: local-testing-multi-document-summarization | ||
|
|
||
| query: | ||
| _source: ["content", "title", "url"] | ||
| query: | ||
| bool: | ||
| should: | ||
| - knn: | ||
| field: content_vector | ||
| query_vector: !Embeddings | ||
| text: !Var query | ||
| config: | ||
| type: "openai" | ||
| engine: "text-embedding-3-small" | ||
| dimensions: 1536 | ||
| k: !Var k | ||
| num_candidates: 100 | ||
| - match: | ||
| content: | ||
| query: !Var query | ||
| boost: 4 | ||
| fuzziness: "AUTO" |
29 changes: 29 additions & 0 deletions
29
cmd/escuse-me/queries/examples/search-summaries-embeddings.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| name: "search-summaries-embeddings" | ||
| short: "Search through summaries using embeddings" | ||
| long: "Search through the summaries using semantic similarity via embeddings" | ||
|
|
||
| flags: | ||
| - name: query | ||
| type: string | ||
| help: "Search text to find similar content" | ||
| required: true | ||
| - name: k | ||
| type: int | ||
| help: "Number of results to return" | ||
| default: 5 | ||
|
|
||
| default-index: local-testing-multi-document-summarization | ||
|
|
||
| query: | ||
| _source: ["content", "title", "url"] | ||
| query: | ||
| knn: | ||
| field: content_vector | ||
| query_vector: !Embeddings | ||
| text: !Var query | ||
| config: | ||
| type: "openai" | ||
| engine: "text-embedding-3-small" | ||
| dimensions: 1536 | ||
| k: !Var k | ||
| num_candidates: 100 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.