Skip to content

Commit

Permalink
[SPARK-22067][SQL] ArrowWriter should use position when setting UTF8S…
Browse files Browse the repository at this point in the history
…tring ByteBuffer

## What changes were proposed in this pull request?

The ArrowWriter StringWriter was setting Arrow data using a position of 0 instead of the actual position in the ByteBuffer.  This was currently working because of a bug ARROW-1443, and has been fixed as of
Arrow 0.7.0.  Testing with this version revealed the error in ArrowConvertersSuite test string conversion.

## How was this patch tested?

Existing tests, manually verified working with Arrow 0.7.0

Author: Bryan Cutler <cutlerb@gmail.com>

Closes apache#19284 from BryanCutler/arrow-ArrowWriter-StringWriter-position-SPARK-22067.
  • Loading branch information
BryanCutler authored and ueshin committed Sep 20, 2017
1 parent ee13f3e commit 718bbc9
Showing 1 changed file with 2 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -234,8 +234,9 @@ private[arrow] class StringWriter(val valueVector: NullableVarCharVector) extend

override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {
val utf8 = input.getUTF8String(ordinal)
val utf8ByteBuffer = utf8.getByteBuffer
// todo: for off-heap UTF8String, how to pass in to arrow without copy?
valueMutator.setSafe(count, utf8.getByteBuffer, 0, utf8.numBytes())
valueMutator.setSafe(count, utf8ByteBuffer, utf8ByteBuffer.position(), utf8.numBytes())
}
}

Expand Down

0 comments on commit 718bbc9

Please sign in to comment.