IPFS storage and search

[stjepangolemac]

Is your feature request related to a problem? Please describe.
I would like to be able to run searches on indexes stored on IPFS.

Describe the solution you'd like
The Storage trait in its current state requires you to implement both read and write functions which is problematic for IPFS. The final IPFS url where the index will be available cannot be known in advance. Potential solution is creating a local index, uploading it to IPFS, then querying it from IPFS directly.

The Storage trait could be broken down into two, StorageRead and StorageWrite. That would allow supporting the IPFS search without creating the index directly on IPFS.

Additional context

• https://github.com/ferristseng/rust-ipfs-api
• we still cannot query byte ranges from rust but that is coming
• It might be possible to build indexes directly on IPFS by pointing to an IPNS hash

[fmassot]

Hi @stjepangolemac, thanks for posting this issue. It reminds me of this PR opened in tantivy.

As for quickwit, it is already possible to create and fill an index locally, then, for example, upload the index directory on S3, and then start the server on an AWS instance with the index URI pointing to the right s3 path. You can thus do the same for an IPFS storage.
The commands new, index, and delete are writing at the index URI, but search and serve are only reading in the directory. To be exact, we can write some cache files locally to avoid fetching some files at each request, the copy_to_file method is used in this case.