[Feature Request] Add a token cutoff

[Reledia]

Hello. Sometimes, some pages I save contains essays or stories that can span aboce 20k+ tokens. This does not only used much more credits/money when doing a request, but it consumes a lot of time (from 4s to >50s) and the prompt get lost in the way.

Would it be possible to add an env variable so we can set the max number of tokens to send in a single request?

[MohamedBassem]

Hi, there's already a hardcoded token cutoff of 1500 tokens (to fit in the context window of some models). But making this configurable makes a ton of sense :) Should be easy to implement and include in the next release. Thanks for the suggestion!

[Reledia]

I'm happy to hear this was already implemented, but I observe a different thing in my logs

My version of hoarder should be the latest available

[MohamedBassem]

Oh wow, that's a lot of tokens. Hmmm, I have a guess. I'm currently approximating tokens to words. So technically speaking I'm truncating the content to 1500 words not tokens. Maybe that's where the difference is coming from? Do you have a link that you are willing to share for me to debug this? Thanks!

[Reledia]

I don't wanna share the actual link because it's very explicit, but it was a chapter of a fanfiction from AO3, so very heavy on words

[MohamedBassem]

Turned out, I had a bug in the content truncating logic. Sending a fix now.

[Reledia]

By looking at the commit, was the truncate function sending words only after the 1500th one?

[MohamedBassem]

Yeah, noob mistake :)

[Reledia]

Dont worry, it happens to everyone :)
But maybe in the next release it would be better to inform users to re-process all previous bookmarks?

[MohamedBassem]

The bug only affects bookmarks with content larger than 1500 though, so it's not all bad. I'll mention it in the release note and let people decide whether they want to re-process or not (given that re-processing can be expensive for those using openai for example).

[Reledia]

Seems like the best solution. Thanks for the time

[MohamedBassem]

Hotfix 0.12.1 released with the fix. Thanks for the report!