Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: parallel search partitions in subindex #3060

Merged
merged 1 commit into from
Oct 29, 2024
+11 −8
Merged

perf: parallel search partitions in subindex #3060

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 11 additions & 8 deletions rust/lance/src/index/vector/ivf/v2.rs
View file
Original file line number Diff line number Diff line change
@@ -22,7 +22,7 @@ use deepsize::DeepSizeOf;
use futures::prelude::stream::{self, StreamExt, TryStreamExt};
use lance_arrow::RecordBatchExt;
use lance_core::cache::FileMetadataCache;
use lance_core::utils::tokio::get_num_compute_intensive_cpus;
use lance_core::utils::tokio::{get_num_compute_intensive_cpus, spawn_cpu};
use lance_core::{Error, Result};
use lance_encoding::decoder::{DecoderPlugins, FilterExpression};
use lance_file::v2::reader::{FileReader, FileReaderOptions};
@@ -436,14 +436,17 @@ impl<S: IvfSubIndex + fmt::Debug + 'static, Q: Quantization + fmt::Debug + 'stat
) -> Result<RecordBatch> {
let part_entry = self.load_partition(partition_id, true).await?;
pre_filter.wait_for_ready().await?;

let query = self.preprocess_query(partition_id, query)?;
let param = (&query).into();
let refine_factor = query.refine_factor.unwrap_or(1) as usize;
let k = query.k * refine_factor;
part_entry
.index
.search(query.key, k, param, &part_entry.storage, pre_filter)

spawn_cpu(move || {
let param = (&query).into();
let refine_factor = query.refine_factor.unwrap_or(1) as usize;
let k = query.k * refine_factor;
part_entry
.index
.search(query.key, k, param, &part_entry.storage, pre_filter)
})
.await
}

fn is_loadable(&self) -> bool {

Unchanged files with check annotations Beta

View file
rust/lance-io/src/object_store.rs
pub fn remove_stream<'a>(
&'a self,
locations: BoxStream<'a, Result<Path>>,
) -> BoxStream<Result<Path>> {

Check warning on line 638 in rust/lance-io/src/object_store.rs

GitHub Actions / linux-build (nightly) 

elided lifetime has a name
self.inner
.delete_stream(locations.err_into::<ObjectStoreError>().boxed())
.err_into::<Error>()
View file
rust/lance-file/src/writer/statistics.rs
fn cast_dictionary_arrays<'a, T: ArrowDictionaryKeyType + 'static>(
arrays: &'a [&'a ArrayRef],
) -> Vec<&Arc<dyn Array>> {

Check warning on line 489 in rust/lance-file/src/writer/statistics.rs

GitHub Actions / linux-build (nightly) 

elided lifetime has a name
arrays
.iter()
.map(|x| x.as_dictionary::<T>().values())
View file
rust/lance-index/src/scalar/label_list.rs
fn search_values<'a>(
&'a self,
values: &'a Vec<ScalarValue>,
) -> BoxStream<Result<RowIdTreeMap>> {

Check warning on line 81 in rust/lance-index/src/scalar/label_list.rs

GitHub Actions / linux-build (nightly) 

elided lifetime has a name
futures::stream::iter(values)
.then(move |value| {
let value_query = SargableQuery::Equals(value.clone());