-
Notifications
You must be signed in to change notification settings - Fork 28.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-48148][CORE] JSON objects should not be modified when read as STRING #46408
Changes from 1 commit
46bdf7d
79e8457
07083fd
2f697da
a5c3761
9a78a8d
114d8a3
e9cb9d2
fc77ed0
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -3865,6 +3865,24 @@ abstract class JsonSuite | |
} | ||
} | ||
} | ||
|
||
test("SPARK-48148: decimal precision is preserved when object is read as string") { | ||
withTempPath { path => | ||
|
||
eric-maynard marked this conversation as resolved.
Show resolved
Hide resolved
|
||
val granularFloat = "-999.99999999999999999999999999999999995" | ||
val jsonString = s"""{"data": {"v": ${granularFloat}}}, {"data": {"v": ${granularFloat}}}]""" | ||
eric-maynard marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
Seq(jsonString).toDF() | ||
.repartition(1) | ||
.write | ||
.text(path.getAbsolutePath) | ||
|
||
val df = spark.read.schema("data STRING").json(path.getAbsolutePath) | ||
|
||
val expected = s"""{"v": ${granularFloat}}""" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can you add more test cases for the following?
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added more tests -- can you clarify the last example and what we expect that to do? It seems like invalid JSON |
||
checkAnswer(df, Seq(Row(expected))) | ||
} | ||
} | ||
} | ||
|
||
class JsonV1Suite extends JsonSuite { | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there an existing API to get the remaining content as string? Also, would it work with multi-line JSON?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was not able to find such an existing API -- there is
JacksonParser.getText
but that appears to simply get the current value if it's a string value.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
wrt. multiline JSON, I have added a test to cover this. It seems that the content reference is not a byte array when using multiline mode.