[SPARK-22848][SQL] Eliminate mutable state from Stack #20035
Conversation
| s""" | ||
| |$code | ||
| |$wrapperClass<InternalRow> ${ev.value} = $wrappedArray; | ||
| """.stripMargin, isNull = "false") |
There was a problem hiding this comment.
This change does not use inline = true at ctx.addMutableState for correct code generation.
|
This PR comes from this discussion. |
|
Test build #85189 has finished for PR 20035 at commit
|
| ev.copy(code = code, isNull = "false") | ||
| ev.copy(code = | ||
| s""" | ||
| |InternalRow[] $rowData = new InternalRow[$numRows]; |
There was a problem hiding this comment.
this creates a large array every time, and I don't think we have data copy issues for generator expressions...
There was a problem hiding this comment.
I see. I did not imagine that numRows is large.
I will revert the code for rowData.
|
Test build #85191 has finished for PR 20035 at commit
|
|
Test build #85194 has finished for PR 20035 at commit
|
|
Test build #85198 has finished for PR 20035 at commit
|
|
Jenkins, retest this please |
|
LGTM |
|
Test build #85223 has finished for PR 20035 at commit
|
|
LGTM |
|
Jenkins, retest this please |
|
I think the test failure is not related to this change, but the ongoing work to upgrade pyarrow. |
|
yea it's failing globally, I'm merging this PR, thanks! |
|
Test build #85237 has finished for PR 20035 at commit
|
|
I ran the test (HDFSMetadataLogSuite) locally. It should be fine as it passes. |
What changes were proposed in this pull request?
This PR eliminates mutable states from the generated code for
Stack.How was this patch tested?
Existing test suites