Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sparse indexing #111

Open
teodosin opened this issue Oct 16, 2024 · 0 comments
Open

Sparse indexing #111

teodosin opened this issue Oct 16, 2024 · 0 comments
Labels
enhancement New feature or request foundational A long-term decision or architectural implementation. Worth the extra effort.

Comments

@teodosin
Copy link
Owner

There's no need to index all items always in a directory. The constraint that Karta can only show nodes in its own database is arbitrary. It should load files, directories as nodes, and only if some karta-only data is set on them, such ass pins, attributes, or connections, should they be indexed in the database. Such sparse indexing will be very important for large directories with hundreds or thousands of files, so the db doesn't get immediately bloated. Also in the future, when file destructuring gets implemented, it will be very useful to link just parts of a file to the wider graph without having to index the whole thing. If all the node data is already implicit in the file or directory, the node wrapper isn't needed.

For the current impl this just means that opening a context should also open the physical files without them having to be indexed. Take care to check for already indexed nodes.

@teodosin teodosin added enhancement New feature or request foundational A long-term decision or architectural implementation. Worth the extra effort. labels Oct 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request foundational A long-term decision or architectural implementation. Worth the extra effort.
Projects
None yet
Development

No branches or pull requests

1 participant