Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SPARK-47781][SQL] Handle negative scale decimals for JDBC data sources
### What changes were proposed in this pull request? SPARK-30252 has disabled the definition of the negative scale for decimals. It has a regression that also disabled reading negative scale decimals from JDBC data sources. **Although there is a legacy config to restore the old behavior**, it seemed neither designed for such a case nor convenient in a data pipeline that extracts negative scale decimals from a database such as Oracle to Parquet files w/o negative scale decimal support. In addition, Postgres has the negative scale decimals support since v15, which was one of the supporters for disabling negative scale decimals on our side. In this PR, we change the schema from `decimal(p,s)` to `decimal(p-s,0)` if s<0. ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? Negative scale decimals have many supporters for rounding the internal parts, such as Oracle, Postgres, etc. - Oracle > Negative scale is the number of significant digits to the left of the decimal point, to but not including the least significant digit. For negative scale the least significant digit is on the left side of the decimal point, because the actual data is rounded to the specified number of places to the left of the decimal point. For example, a specification of (10,-2) means to round to hundreds. - Postgres > Beginning in PostgreSQL 15, it is allowed to declare a numeric column with a negative scale. Then values will be rounded to the left of the decimal point. The precision still represents the maximum number of non-rounded digits. Thus, a column declared as NUMERIC(2, -3) will round values to the nearest thousand and can store values between -99000 and 99000, inclusive. It is also allowed to declare a scale larger than the declared precision. Such a column can only hold fractional values, and it requires the number of zero digits just to the right of the decimal point to be at least the declared scale minus the declared precision. For example, a column declared as ### How was this patch tested? new tests ### Was this patch authored or co-authored using generative AI tooling? no Closes #45956 from yaooqinn/SPARK-47781. Authored-by: Kent Yao <yao@apache.org> Signed-off-by: Kent Yao <yao@apache.org>
- Loading branch information