8371297134 Type promotion between integral and floating types #2173
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Update tests: I deliberately promote to float64 here so we don't lose precision from the int32
…get that just returns a bool To prevent dangerous uses where only the static_cast<bool> of its return value was used, but where the type to be promoted to was not the second argument given to it.
cpp/arcticdb/entity/type_utils.cpp
Outdated
|
|
||
| auto target_size = entity::SizeBits::UNKNOWN_SIZE_BITS; | ||
| auto floating_size = is_floating_point_type(left_type) ? left_size : right_size; | ||
| auto integral_size = is_floating_point_type(left_type) ? right_size : left_size; |
There was a problem hiding this comment.
nit: I think using is_integer_type would be easier to follow.
cpp/arcticdb/entity/type_utils.cpp
Outdated
| } else { | ||
| // Non-numeric target type | ||
| return std::nullopt; | ||
| return false; | ||
| } | ||
| } else if (is_floating_point_type(source_type)) { | ||
| if (is_unsigned_type(target_type) || is_signed_type(target_type)) { |
There was a problem hiding this comment.
It's not part of the change but is a bit more readable
Suggested change
| if (is_unsigned_type(target_type) || is_signed_type(target_type)) { | |
| if (is_integer_type(target_type)) { |
| } | ||
|
|
||
| return target; | ||
| auto target_size = entity::SizeBits(uint8_t(std::max(left_size, right_size)) + 1); |
There was a problem hiding this comment.
I get why this works but it seems a bit hacky and fragile.
There was a problem hiding this comment.
This is just the old logic moved around, so I'd rather not change it
cpp/arcticdb/entity/type_utils.cpp
Outdated
| return (is_integer_type(left) && is_floating_point_type(right)) || (is_floating_point_type(left) && is_integer_type(right)); | ||
| } | ||
|
|
||
| std::optional<entity::TypeDescriptor> common_type_float_integer(const entity::TypeDescriptor& left, const entity::TypeDescriptor& right) { |
There was a problem hiding this comment.
nit: If used only in this file mark it as static
Suggested change
| std::optional<entity::TypeDescriptor> common_type_float_integer(const entity::TypeDescriptor& left, const entity::TypeDescriptor& right) { | |
| static std::optional<entity::TypeDescriptor> common_type_float_integer(const entity::TypeDescriptor& left, const entity::TypeDescriptor& right) { |
cpp/arcticdb/entity/type_utils.cpp
Outdated
| } | ||
| } | ||
|
|
||
| std::optional<entity::TypeDescriptor> common_type_mixed_sign_ints(const entity::TypeDescriptor& left, const entity::TypeDescriptor& right) { |
There was a problem hiding this comment.
Suggested change
| std::optional<entity::TypeDescriptor> common_type_mixed_sign_ints(const entity::TypeDescriptor& left, const entity::TypeDescriptor& right) { | |
| static std::optional<entity::TypeDescriptor> common_type_mixed_sign_ints(const entity::TypeDescriptor& left, const entity::TypeDescriptor& right) { |
… tolerance as a result
vasil-pashov
approved these changes
Feb 20, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To fix the bug reported in 8371297134
I've added logic so that when we merge descriptors we combine:
This is because a 16 bit integer can safely fit in a 32 bit float without loss of precision, whereas a 32 bit integer cannot. We could instead always promote up to float 64 which would be more wasteful but simpler.
Separately, for query builder pipelines I added changes so that when combining a float and an integer we always promote up to float64. This misses the cases where it is actually safe to promote to float32, but is simpler and matches Pandas. This is the cause of the test change in
test_sort_merge.py.I reworked
has_valid_type_promotion, introducing a new functionis_valid_type_promotion_to_targetinstead. This is because thehas_valid_type_promotionsignature was dangerous - it returned an optional type that was often interpreted only as a bool (the actual type inside it was ignored). I've replace the call sites that only tested the bool with calls to the newis_valid_type_promotion_to_targetthat only returns a bool.