pnpm start crashes if there is too much data in knowledge

[c-o-i-n-w-i-t-c-h]

Describe the bug
this bug occurs when putting a large amount of data in the knowledge portion of the character file. when starting eliza for the first time (wiped db first), the console will show that the knowledge is being processed up until a point, and then it will print the word Killed followed by  ELIFECYCLE  Command failed with exit code 137.

if i then run pnpm start again, it skips over the knowledge processing steps and runs normally. so it appears that the remainder of the knowledge items past the point of the initial failure will never be incorporated into the agent.

having tried this multiple times with the same character file and clearing the db each time, i have observed that the point of failure varies widely with each attempt. sometimes it dies while processing the 5th knowledge item. sometimes it dies while processing the 50th item.

To Reproduce
add 100 strings of 10k characters each to knowledge, wipe db, and run pnpm start

i tested this on eliza (main), eliza (0.1.5), and eliza-starter

[odilitime]

Killed usually means you're running out of memory (ram)

[c-o-i-n-w-i-t-c-h]

aha, that would make a lot of sense! could i run this locally and then transfer the database/ .env/ character file to the vps?

[shakkernerd]

Yes, absolutely!
You could have your database hosted anywhere, you just need to set POSTGRES_URL in the env and you are good.

[c-o-i-n-w-i-t-c-h]

thanks so much guys i will try it <3

[shakkernerd]

Knowledge should be more optimal with the changes made to local embedding in #1950