You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I want to sort a list based on custom comparator function that is made out of data fusion expressions
To do a comparison sort we obviously need 2 items from the list, so I created the following schema with 2 identical columns:
let schema = Arc::new(Schema::new(vec![Field::new("col_0", list.data_type().clone(), nullable),Field::new("col_1", list.data_type().clone(), nullable),]));
and then I create a record batch with the schema above for every sort_by call
letmut indices = (0..list.len()).collect::<Vec<_>>();
indices.sort_by(
|&a,&b| {let function_input = RecordBatch::try_new(Arc::clone(&schema),vec![list.slice(a,1), list.slice(b,1)],);// evaluate on the record batch// return `std::cmp::Ordering` value})
as you can see I call try_new a lot of times which does a lot of checks that will evaluate to the same thing over and over again
Describe the solution you'd like
It would be helpful to have a function called new_unchecked that is unsafe and doesn't do any checks
Describe alternatives you've considered
Just calling try_new over and over again
The text was updated successfully, but these errors were encountered:
Is how I would recommend providing a similar interface in a columnar execution environment (spark is not a vectorized engine and instead relies on JIT compilation).
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
I want to sort a list based on custom comparator function that is made out of data fusion expressions
To do a comparison sort we obviously need 2 items from the list, so I created the following schema with 2 identical columns:
and then I create a record batch with the schema above for every
sort_by
callas you can see I call
try_new
a lot of times which does a lot of checks that will evaluate to the same thing over and over againDescribe the solution you'd like
It would be helpful to have a function called
new_unchecked
that isunsafe
and doesn't do any checksDescribe alternatives you've considered
Just calling
try_new
over and over againThe text was updated successfully, but these errors were encountered: