Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make Search Model Configurable on Server #544

Merged
merged 32 commits into from
Nov 16, 2023
Merged

Make Search Model Configurable on Server #544

merged 32 commits into from
Nov 16, 2023

Conversation

debanjum
Copy link
Member

@debanjum debanjum commented Nov 15, 2023 •
edited
Loading

  • Make search model configurable on server
  • Update migration script to get search model from khoj.yml to Postgres
  • Update first run message on Khoj Desktop and Web app landing page
  • Other miscellaneous bug fixes

@debanjum
Make search model configurable on server
4af194d 
- Expose ability to modify search model via Django admin interface
- Previously the bi_encoder and cross_encoder models to use were set
  in code
- Now it's user configurable but with a default config generated by
  default
@debanjum
Add search model config from khoj.yml to Postgres DB via migration sc…
3d8d614 
…ript
@debanjum
Only create KhojUser on login if doesn't already exist
f8e5e11
@debanjum
Remove unused parameter from configure_search_type method
8f200cf
@debanjum debanjum force-pushed the make-search-models-configurable branch from 8af12db to 8f200cf Compare November 15, 2023 03:09
sabaimran
sabaimran approved these changes Nov 15, 2023
View reviewed changes
Copy link
Member

@sabaimran sabaimran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few comments, but looks good!

sabaimran
sabaimran requested changes Nov 15, 2023
View reviewed changes
sabaimran and others added 11 commits November 15, 2023 17:03
@sabaimran
Update format for return type of the generate token mehtod
8e62af7
@sabaimran
Allow dockerize step to run for prod from PR temporarily
cf74fa4
@sabaimran
Update format for return type of the generate token method
154de8c
@sabaimran @debanjum
Update logic for creating a new user to use aupdate_or_create
bbae7dd
@sabaimran @debanjum
Add a dev workflow for GitHub actions, change the production workflow…
70f5d0e 
… to only kick off when pushed to master
@sabaimran @debanjum
Rename dockerize dev action to be more accurate
10be8df
@sabaimran @debanjum
Fix return type of the update_or_create method
245a9cb
@sabaimran @debanjum
Allow dockerize step to run for prod from PR temporarily
f88a586
@debanjum
Use embeddings model store from state in text to entries
0679b2a 
Do not need to instantiating it separately. In all other places we're
using the embeddings model store in global state anyway
@debanjum
Rename SearchModel to SearchModelConfig DB model, Require Cross-Encoder
08a057b
@debanjum
Use better name for DB adapter func to create user by Google token
348cc0c
@debanjum debanjum force-pushed the make-search-models-configurable branch from cf74fa4 to 348cc0c Compare November 16, 2023 01:32
sabaimran and others added 7 commits November 15, 2023 17:32
@sabaimran
Resolve merge conflicts in auth.py with remove KhojApiUser import
6b17aeb
@sabaimran
Merge with master
ea144de
@sabaimran
Merge pull request #547 from khoj-ai/features/fix-api-token-generator
0da4db4 
Update the return type of the API token generator
@debanjum
Use Sigmoid to normalize cross-encoder score between 0-1
18dbad5 
- While sigmoid normalization isn't required for reranking.
  Normalizing score to distance metrics for both encoder and cross
  encoder scores is useful to reason about them
- Softmax wasn't required as don't need probabilities, sigmoid is good
  enough to get distance metric
@debanjum
Set max cos distance to 0.18. Test search API query with max distance
922983b
@debanjum
Update first run message on Desktop app with API token setup instruct…
6c1693b 
…ions

- Open Web app settings in the default browser via link click
- Open Desktop app settings via link click
@debanjum
Update first run message on Web app with Chat models setup instructions
cc05013 
- Link to Django admin panel for user to create Chat Models on their
  Khoj server
- This should only get hit when user is not using Khoj cloud, as Khoj
  cloud would already have Chat models configured
@debanjum debanjum merged commit 208dddd into features/multi-user-support-khoj Nov 16, 2023
@debanjum debanjum deleted the make-search-models-configurable branch November 16, 2023 08:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sabaimran sabaimran sabaimran requested changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@debanjum @sabaimran