Skip to content

Commit

Permalink
[SPARK-39979][SQL][FOLLOW-UP] Make `spark.sql.execution.arrow.useLarg…
Browse files Browse the repository at this point in the history
…eVarTypes` as an internal configuration

### What changes were proposed in this pull request?

This PR is a followup of apache#39572 that hides the `spark.sql.execution.arrow.useLargeVarTypes` configuration as an internal configuration.

### Why are the changes needed?

As described in apache#41569, this feature only works for `mapInArrow`, and other cases cannot be completely supported because of Arrow side limitation, see apache/arrow#35289. Therefore, this PR hides this configuration as an internal one for now.

### Does this PR introduce _any_ user-facing change?

No, this configuration was not released out yet.

### How was this patch tested?

Ran the Scala linter.

Closes apache#41584 from HyukjinKwon/SPARK-39979-followup2.

Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
HyukjinKwon authored and czxm committed Jun 19, 2023
1 parent 71663da commit 2fa347c
Showing 1 changed file with 2 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -2841,8 +2841,9 @@ object SQLConf {
.doc("When using Apache Arrow, use large variable width vectors for string and binary " +
"types. Regular string and binary types have a 2GiB limit for a column in a single " +
"record batch. Large variable types remove this limitation at the cost of higher memory " +
"usage per value.")
"usage per value. Note that this only works for DataFrame.mapInArrow.")
.version("3.5.0")
.internal()
.booleanConf
.createWithDefault(false)

Expand Down

0 comments on commit 2fa347c

Please sign in to comment.