Skip to content

Commit

Permalink
Implement Spark decimal add and subtract (facebookincubator#5791)
Browse files Browse the repository at this point in the history
Summary:
Use Arrow Gandiva BasicDecimal128 algorithm to compute value.
Arrow implementation:
https://github.com/apache/arrow/blob/release-12.0.1-rc1/cpp/src/gandiva/precompiled/decimal_ops.cc#L211-L231

Spark result precision and scale maybe different with Presto, because it will use adjustPrecisionScale to change the precision and scale when precision is beyond 38.
And this implement can compute data without overflow in some situation.

Pull Request resolved: facebookincubator#5791

Reviewed By: kgpai, pedroerp

Differential Revision: D52220210

Pulled By: Yuhta

fbshipit-source-id: 96f9be2c36d37d11aa71df4308a6a0c9a3bd50a3
  • Loading branch information
jinchengchenghh authored and facebook-github-bot committed Dec 18, 2023
1 parent 97c273b commit d3f0dc9
Show file tree
Hide file tree
Showing 8 changed files with 684 additions and 78 deletions.
26 changes: 26 additions & 0 deletions velox/docs/functions/spark/math.rst
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,20 @@ Mathematical Functions
Returns the result of adding x to y. The types of x and y must be the same.
For integral types, overflow results in an error. Corresponds to sparks's operator ``+``.

.. spark:function:: add(x, y) -> decimal
Returns the result of adding ``x`` to ``y``. The argument types should be DECIMAL, and can have different precisions and scales.
Fast path is implemented for cases that should not overflow. For the others, the whole parts and fractional parts of input decimals are added separately and combined finally.
The result type is calculated with the max precision of input precisions, the max scale of input scales, and one extra digit for possible carrier.
Overflow results in null output. Corresponds to Spark's operator ``+``.

::

SELECT CAST(1.1232100 as DECIMAL(38, 7)) + CAST(1 as DECIMAL(10, 0)); -- DECIMAL(38, 6) 2.123210
SELECT CAST(-999999999999999999999999999.999 as DECIMAL(30, 3)) + CAST(-999999999999999999999999999.999 as DECIMAL(30, 3)); -- DECIMAL(31, 3) -1999999999999999999999999999.998
SELECT CAST(99999999999999999999999999999999.99998 as DECIMAL(38, 6)) + CAST(-99999999999999999999999999999999.99999 as DECIMAL(38, 5)); -- DECIMAL(38, 6) -0.000010
SELECT CAST(-99999999999999999999999999999999990.0 as DECIMAL(38, 3)) + CAST(-0.00001 as DECIMAL(38, 7)); -- DECIMAL(38, 6) NULL

.. spark:function:: bin(x) -> varchar
Returns the string representation of the long value ``x`` represented in binary.
Expand Down Expand Up @@ -179,6 +193,18 @@ Mathematical Functions
Returns the result of subtracting y from x. The types of x and y must be the same.
For integral types, overflow results in an error. Corresponds to Spark's operator ``-``.

.. spark:function:: subtract(x, y) -> decimal
Returns the result of subtracting ``y`` from ``x``. Reuses the logic of add function for decimal type.
Corresponds to Spark's operator ``-``.

::

SELECT CAST(1.1232100 as DECIMAL(38, 7)) - CAST(1 as DECIMAL(10, 0)); -- DECIMAL(38, 6) 0.123210
SELECT CAST(-999999999999999999999999999.999 as DECIMAL(30, 3)) - CAST(-999999999999999999999999999.999 as DECIMAL(30, 3)); -- DECIMAL(31, 3) 0.000
SELECT CAST(99999999999999999999999999999999.99998 as DECIMAL(38, 6)) - CAST(-0.00001 as DECIMAL(38, 5)); -- DECIMAL(38, 6) 99999999999999999999999999999999.999990
SELECT CAST(-99999999999999999999999999999999990.0 as DECIMAL(38, 3)) - CAST(0.00001 as DECIMAL(38, 7)); -- DECIMAL(38, 6) NULL

.. spark:function:: unaryminus(x) -> [same as x]
Returns the negative of `x`. Corresponds to Spark's operator ``-``.
Loading

0 comments on commit d3f0dc9

Please sign in to comment.