AI chatting functionality #11430

InAnYan · 2024-06-26T06:20:57Z

Screenshots of the UI changes

Old chat style:

New chat style:

Summarization:

AI preferences:

TODOs

(Update by koppor)

All issues of Milestone https://github.com/InAnYan/jabref/milestone/1 need to be fixed
All issues of Milestone https://github.com/InAnYan/jabref/milestone/2 need to be fixed
All issues of Milestone https://github.com/InAnYan/jabref/milestone/3 need to be fixed
Review comments adressed

Mandatory checks

Change in CHANGELOG.md described in a way that is understandable for the average user (if applicable)
Tests created for changes (if applicable)
Manually tested changed features in running JabRef (always required)
Screenshots added in PR description (for UI changes)
Checked developer's documentation: Is the information available and up to date? If not, I outlined it in this pull request.
Checked documentation: Is the information available and up to date? If not, I created an issue at https://github.com/JabRef/user-documentation/issues or, even better, I submitted a pull request to the documentation repository.

InAnYan · 2024-06-26T06:21:41Z

On my machine, when I run "checkstyleMain" from Gradle commands, I have no such errors. It has successfully generated for me a list of violations

Siedlerchr · 2024-06-26T10:34:44Z

Then it seems reviewdog seems to have an outdated version of checkstyle

src/main/java/org/jabref/logic/ai/embeddings/EmbeddingsGenerationTaskManager.java

src/main/java/org/jabref/logic/ai/chathistory/BibDatabaseChatHistory.java

Siedlerchr · 2024-06-26T18:12:15Z

I updated the checkstyle reviewdog action to a newer version so the checkstyle works now again and can parse your code :)

src/main/java/org/jabref/gui/preferences/ai/AiTab.java

koppor · 2024-06-27T09:16:49Z

Blocked by Cannot debug JabRef on Windows in IntelliJ IDEA InAnYan/jabref#5
Blocked by IntelliJ IDEA requires modules for langchain4j InAnYan/jabref#6

calixtus

Just some quick notes after i have skimmed the changes.

calixtus · 2024-07-01T13:56:06Z

src/main/java/org/jabref/logic/pdf/search/IndexingTaskManager.java

+
+        this.onFailure(e -> {
+            throw new RuntimeException(e);
+        });


Be aware that the whole indexing architecture will be refactored when the lucene search project is integrated.
This will produce merge conflicts. The less changes in the search package, the better.

I think, this change can be reverted.

NEVER use new RuntimeException(e);. This tears down the whole application and all data is lost. Is this error really that severe? No, it is not.

calixtus · 2024-07-01T13:57:43Z

src/main/java/org/jabref/logic/pdf/search/PdfIndexer.java

@@ -42,6 +42,7 @@

 /**
 * Indexes the text of PDF files and adds it into the lucene search index.
+ * Also generates embeddings.


Will cause merge conflicts when lucene search is merged.

I think, the whole change can be reverted.

I clicked the revert button.

calixtus · 2024-07-01T14:02:56Z

src/main/java/org/jabref/gui/Base.css

+
+    /* AI chat style */
+    -jr-ai-message-user: -jr-accent;
+    -jr-ai-message-ai: -jr-accent;
+
+    /* MarkdownViewer settings */
+    -mdfx-font-color: black;
+    -mdfx-link-color: blue;
+    -mdfx-border-color-1: #888;
+
+    -mdfx-bg-color-1: #ccc;
+    -mdfx-bg-color-2: #ddd;
+    -mdfx-bg-color-3: #eee;
+
+    -mdfx-bq-color-border: #4488cc; /* the blockquote's border color */
+    -mdfx-bq-color-background: #0000ff0c; /* the blockquote's background color */
+}
+
+.markdown-text {
+    -fx-font-family: "System";
+    -fx-font-size: 13;
+}
+
+.markdown-italic {
+    -fx-font-style: italic;
+}
+
+.markdown-bold {
+    -fx-font-weight: bold;
+}
+
+.markdown-strikethrough {
+    -fx-strikethrough: true;


Seems to be an artifact after ADR34

That mean,s these .markdown-* and maybe -mdfx-* items can be removed.

src/main/java/org/jabref/gui/ai/components/privacynotice/PrivacyNoticeComponent.java

koppor · 2024-07-04T15:57:24Z

The embeddings generation makes my machine unsuable. Please try to use lower priority for the indexing threads:

Without embeddings generation:

ThiloteE · 2024-07-04T16:14:42Z

Use as many threads as there are physical cores. Using LLMs, that's the most performant in my personal tests.

koppor

I went half thorugh. Will continue later

General comment:

NEVER use new RuntimeException(e);. This tears down the whole application and all data is lost. Is this error really that severe? No, it is not.

Minr comments:

I reverted changes in files which have no impact on the concrete functionality.

I also reverted level@org.jabref.logic.ai.embeddings.AiIngestor = trace, because this is for development only, not for a distribution of JabRef.

I also made code suggestions. You can collect them and commit them togehter (and not create several commits) - https://docs.github.com/de/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/incorporating-feedback-in-your-pull-request

koppor · 2024-07-04T12:11:42Z

src/main/java/org/jabref/gui/Base.css

+
+    /* AI chat style */
+    -jr-ai-message-user: -jr-accent;
+    -jr-ai-message-ai: -jr-accent;
+
+    /* MarkdownViewer settings */
+    -mdfx-font-color: black;
+    -mdfx-link-color: blue;
+    -mdfx-border-color-1: #888;
+
+    -mdfx-bg-color-1: #ccc;
+    -mdfx-bg-color-2: #ddd;
+    -mdfx-bg-color-3: #eee;
+
+    -mdfx-bq-color-border: #4488cc; /* the blockquote's border color */
+    -mdfx-bq-color-background: #0000ff0c; /* the blockquote's background color */
+}
+
+.markdown-text {
+    -fx-font-family: "System";
+    -fx-font-size: 13;
+}
+
+.markdown-italic {
+    -fx-font-style: italic;
+}
+
+.markdown-bold {
+    -fx-font-weight: bold;
+}
+
+.markdown-strikethrough {
+    -fx-strikethrough: true;


That mean,s these .markdown-* and maybe -mdfx-* items can be removed.

src/main/java/org/jabref/gui/Dark.css

src/main/java/org/jabref/gui/JabRefGUI.java

src/main/java/org/jabref/logic/ai/chat/AiChatLanguageModel.java

src/main/java/org/jabref/logic/ai/embeddings/AiEmbeddingsManager.java

src/main/java/org/jabref/logic/ai/embeddings/AiEmbeddingModel.java

koppor · 2024-07-04T19:35:27Z

If embedding generation is cancleded, JabRef does not shut down cleanly:

Meaning: It does not shut down at all.

koppor · 2024-07-04T20:38:26Z

If no libraries are opened (e..,g because they were closed), no indexing should happen

koppor · 2024-07-04T20:55:54Z

Please also test with a shared SQL database. See https://docs.jabref.org/collaborative-work/sqldatabase for details.

Ensure that autosave to a local .tex file is activated.

i assume that either a check for database location == LOCAL -has to be added -- or another storage path for the chat history used.

koppor · 2024-07-04T21:12:47Z

Good thing: seems to count files right.

@Article{test,
  file = {:test.pdf},
}
@Comment{jabref-meta: databaseType:bibtex;}

koppor · 2024-07-04T21:14:35Z

Bad thing: if test.pdf was made availble using the file system (and automatic relinking by JAbRef), it is not indexed:

To reproduce: Use BibTeX from above. Open JabRef. Check indexing. Then copy test.pdf to a folder where JabRef can find the PDF.

koppor

Last comments :)

Not big ones, just some code things so that others could understand it more if they were about to maintain it.

I am worried about the shutdown. To get this fixed, I think, we need to add LOGGER.trace statements at org.jabref.gui.LibraryTab#onClosed and other places where a shutdown happens - to see which shutdowns are triggered and which are not happening.

src/main/java/org/jabref/gui/JabRefGUI.java

src/main/java/org/jabref/gui/LibraryTab.java

src/main/java/org/jabref/gui/ai/RegenerateEmbeddingsAction.java

src/main/java/org/jabref/gui/entryeditor/AiChatTab.java

src/main/java/org/jabref/logic/ai/chathistory/BibDatabaseChatHistory.java

koppor · 2024-07-04T21:32:43Z

src/main/java/org/jabref/gui/entryeditor/EntryEditor.java

-        tabs.add(new DeprecatedFieldsTab(databaseContext, libraryTab.getSuggestionProviders(), undoManager, dialogService, preferencesService, stateManager, themeManager, libraryTab.getIndexingTaskManager(), bibEntryTypesManager, taskExecutor, journalAbbreviationRepository));
-        tabs.add(new OtherFieldsTab(databaseContext, libraryTab.getSuggestionProviders(), undoManager, dialogService, preferencesService, stateManager, themeManager, libraryTab.getIndexingTaskManager(), bibEntryTypesManager, taskExecutor, journalAbbreviationRepository));
+        tabs.add(new RequiredFieldsTab(databaseContext, libraryTab.getSuggestionProviders(), undoManager, dialogService, preferencesService, stateManager, themeManager, libraryTab.getIndexingTaskManager(), libraryTab.getEmbeddingsGenerationTaskManager(), bibEntryTypesManager, taskExecutor, journalAbbreviationRepository));
+        tabs.add(new ImportantOptionalFieldsTab(databaseContext, libraryTab.getSuggestionProviders(), undoManager, dialogService, preferencesService, stateManager, themeManager, libraryTab.getIndexingTaskManager(), libraryTab.getEmbeddingsGenerationTaskManager(), bibEntryTypesManager, taskExecutor, journalAbbreviationRepository));


The parameter libraryTab.getEmbeddingsGenerationTaskManager( is required for the AiTab only. Remove everywhere else.

No, it's required for PreviewPanel or PreviewTab (#11430 (comment))

ImportantOptionalFieldsTab do not have a preivew. Therefore, passing the whole EmbeddingsGenerationTaskManager is strange.

@calixtus A) Pass EmbeddingsGenerationTaskManager down to all tabs or B) use ´Injector.instantiateModelOrService(EmbeddingsGenerationTaskManager.class)` when really needed?

comment to A): This is required, because we have abstract class FieldsEditorTab extends EntryEditorTab implements OffersPreview {. And each preview offers drag'n'drop of files. Thus, file events should be handled.

I see more options (with more work)

C): Move implements OffersPreview to the tabs really offering a preview and try to pass the EmbeddingsGenerationTaskManager only to those

D) Use "the" event bus to handle file events and have EmbeddingsGenerationTaskManager subscribing to these file changes. This way, we would decouple the places and have "only" org.jabref.gui.externalfiles.ExternalFilesEntryLinker#moveFilesToFileDirRenameAndAddToEntry knowing the task managers - and not somewhere in the UI.

Option D seems very interesting)

Option D seems very interesting)

I like most. But one needs to take care to jump from the JavaFX thread to a Background Thread.

Hints:

https://www.reddit.com/r/JavaFX/s/9QQOHKKbKb

https://github.com/google/guava/blob/master/android/guava/src/com/google/common/util/concurrent/WrappingExecutorService.java

I think, we did that. Thus. closing this.

src/main/java/org/jabref/gui/entryeditor/EntryEditor.java

koppor · 2024-07-04T21:37:37Z

Use as many threads as there are physical cores. Using LLMs, that's the most performant in my personal tests.

Maybe, we need a setting. If the thing takes 1 hour on a "normal" machine to create embeddings and the user cannot work in parallel, this is a no-go in my oppinion. The PC should be usable without any signifcant lags.

I know that Linux is a better in multi tasking. However, there are still persons using Windows. Thus, JabRef should also be useable for Windows users.

koppor

Some more comments maybe (foudn another firefox window)

src/main/java/org/jabref/preferences/JabRefPreferences.java

src/main/java/org/jabref/gui/preferences/ai/AiTabViewModel.java

src/main/java/org/jabref/gui/entryeditor/aichattab/AiChatTabWorking.java

src/main/java/org/jabref/logic/ai/embeddings/MVStoreEmbeddingStore.java

src/main/java/org/jabref/logic/ai/chat/AiChatLanguageModel.java

koppor · 2024-08-13T22:06:38Z

I refactored API key handling - but org.jabref.gui.menus.FileHistoryMenuTest still has mock issues.

InAnYan · 2024-08-14T08:17:38Z

@koppor #11430 (comment)

what kind of mock issues?

I ran the test, and it seems okay, or am I doing something wrong?

Siedlerchr · 2024-08-14T09:06:45Z

yeah seems to be all green now

koppor · 2024-08-14T09:19:39Z

@koppor #11430 (comment)

what kind of mock issues?

I ran the test, and it seems okay, or am I doing something wrong?

Revert my commit 8ec3876 and try again. (Do not push the revert)

git revert 8ec3876

koppor · 2024-08-14T09:21:40Z

For me, it is OK. More changes can be done in a follow-up PR. We collected issues for follow up at https://github.com/InAnYan/jabref/milestone/12.

koppor · 2024-08-14T11:25:37Z

I think, the last commit is OK.

Even if I fear that changing the prompt only will lead to a complete re-indexing, which is not desired at all!

This reverts commit 5fa3bb5.

github-actions · 2024-08-14T14:21:09Z

The build for this PR is no longer available. Please visit https://builds.jabref.org/main/ for the latest build.

subhramit · 2024-08-14T17:42:24Z

Congratulations, Ruslan. This was a big one (for you as well as for JabRef)!

github-actions bot reviewed Jun 26, 2024

View reviewed changes

src/main/java/org/jabref/logic/ai/embeddings/EmbeddingsGenerationTaskManager.java Outdated Show resolved Hide resolved

src/main/java/org/jabref/logic/ai/chathistory/BibDatabaseChatHistory.java Outdated Show resolved Hide resolved

github-actions bot reviewed Jun 27, 2024

View reviewed changes

src/main/java/org/jabref/gui/preferences/ai/AiTab.java Outdated Show resolved Hide resolved

koppor mentioned this pull request Jun 27, 2024

Cannot debug JabRef on Windows in IntelliJ IDEA InAnYan/jabref#5

Closed

2 tasks

InAnYan added the project: gsoc label Jun 27, 2024

calixtus reviewed Jul 1, 2024

View reviewed changes

koppor changed the title ~~AI chatting functionality (the right branch)~~ AI chatting functionality Jul 3, 2024

koppor requested changes Jul 4, 2024

View reviewed changes

koppor added this to the 6.0-alpha milestone Jul 4, 2024

koppor mentioned this pull request Jul 4, 2024

Update to JDK22 for building #11057

Closed

7 tasks

koppor requested changes Jul 4, 2024

View reviewed changes

koppor reviewed Jul 4, 2024

View reviewed changes

ThiloteE mentioned this pull request Jul 5, 2024

Let embedding model run on GPU InAnYan/jabref#71

Open

InAnYan and others added 4 commits July 6, 2024 12:37

Fix the code from code review

8ca536e

Fix from code review and create new AiChatTabWorking

1e7221a

Improve chat history storage code

874e513

More fix from code review

a3b9dd9

github-actions bot reviewed Jul 8, 2024

View reviewed changes

Remove obsolete parameter

2d65495

koppor added 4 commits August 13, 2024 22:31

Fix localization to be more provider independent

e18eda9

Fix method names

d3e8155

Add poor man's solution to notify of API key changes

26e279a

Reduce calls to key store (and fix key saving)

d4a96d0

Merge branch 'main' into ai-pr-1

3c8cb68

koppor previously approved these changes Aug 13, 2024

View reviewed changes

Fix for InAnYan#148 and partially InAnYan#146

5fa3bb5

InAnYan dismissed koppor’s stale review via 5fa3bb5 August 14, 2024 10:53

Revert "Fix for InAnYan#148 and partially InAnYan#146"

58bdb8c

This reverts commit 5fa3bb5.

koppor previously approved these changes Aug 14, 2024

View reviewed changes

Fix for scrolling down when deleting a message

f794012

InAnYan dismissed koppor’s stale review via f794012 August 14, 2024 12:33

InAnYan added 4 commits August 14, 2024 16:08

Sort EmbeddingModel enum variants

9730fe8

Fix GenerateSummaryTask progress indication

d0e7170

Fix dark mode

8aa5fbb

Add notice for embedding models size

ddd979a

Siedlerchr enabled auto-merge August 14, 2024 16:01

Siedlerchr approved these changes Aug 14, 2024

View reviewed changes

Siedlerchr added this pull request to the merge queue Aug 14, 2024

Merged via the queue into main with commit 8a0edc2 Aug 14, 2024
27 of 28 checks passed

Siedlerchr deleted the ai-pr-1 branch August 14, 2024 16:14

AI chatting functionality #11430

AI chatting functionality #11430

Conversation

InAnYan commented Jun 26, 2024 • edited Loading

Screenshots of the UI changes

TODOs

Mandatory checks

InAnYan commented Jun 26, 2024

Siedlerchr commented Jun 26, 2024

Siedlerchr commented Jun 26, 2024

koppor commented Jun 27, 2024 • edited by InAnYan Loading

calixtus left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

koppor commented Jul 4, 2024 • edited Loading

ThiloteE commented Jul 4, 2024 • edited Loading

koppor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

koppor commented Jul 4, 2024 • edited Loading

koppor commented Jul 4, 2024

koppor commented Jul 4, 2024

koppor commented Jul 4, 2024

koppor commented Jul 4, 2024

koppor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

koppor commented Jul 4, 2024

koppor left a comment

Choose a reason for hiding this comment

koppor commented Aug 13, 2024

InAnYan commented Aug 14, 2024

Siedlerchr commented Aug 14, 2024

koppor commented Aug 14, 2024

koppor commented Aug 14, 2024

koppor commented Aug 14, 2024

github-actions bot commented Aug 14, 2024 • edited Loading

subhramit commented Aug 14, 2024

InAnYan commented Jun 26, 2024 •

edited

Loading

koppor commented Jun 27, 2024 •

edited by InAnYan

Loading

koppor commented Jul 4, 2024 •

edited

Loading

ThiloteE commented Jul 4, 2024 •

edited

Loading

koppor commented Jul 4, 2024 •

edited

Loading

github-actions bot commented Aug 14, 2024 •

edited

Loading