Inference Failed with Ollama

[joshuapeterson]

I added ollama support per instructions. Some of the requests seemed to work fine but I found the following in the logs:
2024-03-27T14:01:40.649Z error: [inference][17] inference job failed: Error: Expected a completed response. 2024-03-27T14:06:40.997Z error: [inference][18] inference job failed: TypeError: fetch failed

At least one of my bookmarks does not have any tags added to it.

I'm running this on a docker container on linux.

[MohamedBassem]

Ollama is a new addition (a couple of hours old), so it's not as battle tested as using openAI. I'll probably add Experimental beside it in the documentation.

I managed to reproduce some weird behaviors at least with the llama2 model locally. Sometimes it doesn't ignores the instructions and outputs random jsons, etc. I'll try to dig deeper in what's happening. Thanks for the report!

[MohamedBassem]

Added some warnings in 5cbce67.

[joshuapeterson]

For sure these are not high priority issues. Just trying to add issues as I see them to help with your development. Liking what I'm seeing so far.

I just got a new one that had more data in the response:
2024-03-27T14:52:32.188Z error: [inference][25] inference job failed: Error: [inference][25] Failed to parse JSON response from inference client: [ { "code": "invalid_type", "expected": "array", "received": "undefined", "path": [ � "tags" ], "message": "Required" } ]

[MohamedBassem]

Keep the bug reports coming, they are really helpful!

Yes, that's the error I got as well. As I mentioned, the model sometimes decides to ignore my instruction and outputs either an invalid json or a random json :D This is what casues this error. Btw, I'm curious, which model are you using? llama2?

[joshuapeterson]

Yes, using llama2. No other reason except that's what was in your documentation. I've used Ollama for about 2 hours at this point... :)

[MohamedBassem]

I sent 8156e8c to solve the Expected a completed response problem in ollama, but the model still sometimes outputs JSONs not confronting to the format I requested. So it'll have a better success rate than what it used to have, but still not 100% as that of openai.

[dyslexicjedi]

Not to jump onto your thread. But I've had two issues with this as well.

First:
2024-03-28T00:23:22.223Z info: [Crawler][12] Will crawl "https://css-tricks.com/creating-an-editable-textarea-that-supports-syntax-highlighted-code/" for link with id "g6avp8icpxw0w9q4tiyjl2sk"
2024-03-28T00:23:22.233Z info: [search][27] Completed successfully
2024-03-28T00:23:26.825Z info: [Crawler][12] Completed successfully
2024-03-28T00:23:26.838Z info: [search][28] Completed successfully
2024-03-28T00:23:31.077Z error: [inference][14] inference job failed: Error: [inference][14] Failed to parse JSON response from inference client: SyntaxError: Bad control character in string literal in JSON at position 243 (line 3 column 162)

Second:
2024-03-28T00:24:49.026Z info: [Crawler][13] Will crawl "https://docs.docker.com/engine/install/ubuntu/" for link with id "mjucq8li26nhh71ll97pf8id"
2024-03-28T00:24:49.033Z info: [search][29] Completed successfully
2024-03-28T00:24:51.975Z info: [Crawler][13] Completed successfully
2024-03-28T00:24:51.982Z info: [search][30] Completed successfully
2024-03-28T00:24:54.865Z error: [inference][15] inference job failed: Error: [inference][15] Failed to parse JSON response from inference client: [
{
"code": "invalid_type",
"expected": "array",
"received": "undefined",
"path": [
"tags"
],
"message": "Required"
}
]

[MohamedBassem]

@dyslexicjedi yeah, that one is the model not respecting the prompt. Sometimes the models just forgets about the prompt (I guess with long contexts?) and starts just outputing random JSONs. I wonder if there are some techniques that I can use for the model not to lose context. For example, Maybe I share the content first, and then ask the model to give me the tags? 🤔

[kamtschatka]

@dyslexicjedi Seems like I had the same idea as MohamedBassem. I tried it out before reading his answer and it seems to work fine.

Basically I split the existing prompt into 2 and added the "respond with json and 3-5 tags" part at the end:
031f025

For the samples above, the llama3 model responded properly:

• "Install Docker Engine on Ubuntu": docker, installation, linux
• "Creating an Editable Textarea That Supports Syntax-Highlighted Code | CSS-Tricks": code editor, javascript, css, html, syntax highlighting

I also tried it with the ~20 links I have in my hoarder instance so far and all of them were tagged (previously 3 weren't).
So if anybody has more links that did not work, please let me know, so I can try it out.

[MohamedBassem]

@kamtschatka the change you shared in general looks good to me. I'm happy to accept it in a PR if this improves ollama's accuracy :) I can also give it a try a bit with openAI.

[roadkingvrod]

Hi,

I just set up Hoarder with OLLAMA. When it runs the inference jobs, i get errors such as:

2024-05-17T18:13:08.088Z info: [inference][68] Starting an inference job for bookmark with id "dzrjut9qb81couvfepd5pr7g"
2024-05-17T18:13:08.090Z info: [search][134] Attempting to index bookmark with id dzrjut9qb81couvfepd5pr7g ...
2024-05-17T18:13:08.099Z error: [inference][68] inference job failed: TypeError: fetch failed
2024-05-17T18:13:08.162Z info: [search][134] Completed successfully

I don't know where else to look for deeper log data, so any input is greatly appreciated. I'm running the models: llama3 and llava-llama3 for image.

Thanks!

[kamtschatka]

Sounds like it is not reachable, or running on the wrong port or something like that.
When I change the port to something that is incorrect i get the same output:

2024-05-17T18:20:01.359Z info: [inference][603] Starting an inference job for bookmark with id "lcru3ue290smw8lfxi74llny"
2024-05-17T18:20:03.400Z error: [inference][603] inference job failed: TypeError: fetch failed

[roadkingvrod]

Sounds like it is not reachable, or running on the wrong port or something like that. When I change the port to something that is incorrect i get the same output:

2024-05-17T18:20:01.359Z info: [inference][603] Starting an inference job for bookmark with id "lcru3ue290smw8lfxi74llny"
2024-05-17T18:20:03.400Z error: [inference][603] inference job failed: TypeError: fetch failed

Thank you! That gave me enough direction to figure out what was going on. Getting successes now.

[roadkingvrod]

As I continue to use the app with Ollama, I am having good success with the text based tagging using the llama3 stack. However, I am not having any of the image searches work, and they all create failed inferences. I've tried with llava and llava-llama3.

I'm getting these errors in the workers log, but am at a loss to further troubleshoot. Any suggestions?

2024-05-22T16:23:33.588Z info: [inference][355] Starting an inference job for bookmark with id "z0t1rbmw8pslkc8rcjsp24zt"
2024-05-22T16:23:37.713Z error: [inference][355] inference job failed: Error: [inference][355] The model ignored our prompt and didn't respond with the expected JSON: {"issues":[{"code":"invalid_type","expected":"array","received":"undefined","path":["tags"],"message":"Required"}],"name":"ZodError"}. Here's a sneak peak from the response: {"automatic tagging"

[roadkingvrod]

As I continue to use the app with Ollama, I am having good success with the text based tagging using the llama3 stack. However, I am not having any of the image searches work, and they all create failed inferences. I've tried with llava and llava-llama3.

I'm getting these errors in the workers log, but am at a loss to further troubleshoot. Any suggestions?

2024-05-22T16:23:33.588Z info: [inference][355] Starting an inference job for bookmark with id "z0t1rbmw8pslkc8rcjsp24zt" 2024-05-22T16:23:37.713Z error: [inference][355] inference job failed: Error: [inference][355] The model ignored our prompt and didn't respond with the expected JSON: {"issues":[{"code":"invalid_type","expected":"array","received":"undefined","path":["tags"],"message":"Required"}],"name":"ZodError"}. Here's a sneak peak from the response: {"automatic tagging"

Anything I can do to help troubleshoot this?

[kamtschatka]

Provide instructions on what you are doing/have configured and a sample image.
Are you on the latest version of everything?

I just tried to upload an image and got proper tagging, so it can't be a general issue.

[MohamedBassem]

It seems that ollama has been mostly stable with folks so far. So I'm closing this for now.