Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-7565][SQL] Also consider java string in writePrimitive for StringType #6084

Closed
wants to merge 2 commits into from
Closed

[SPARK-7565][SQL] Also consider java string in writePrimitive for StringType #6084

wants to merge 2 commits into from

Conversation

viirya
Copy link
Member

@viirya viirya commented May 12, 2015

@SparkQA
Copy link

SparkQA commented May 12, 2015

Test build #32509 has finished for PR 6084 at commit cc3ec61.

  • This patch fails Python style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented May 12, 2015

Test build #32514 has finished for PR 6084 at commit d666916.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@JoshRosen
Copy link
Contributor

/cc @davies @yhuai, it looks like this might be fixing a 1.4.0 regression, since UTF8String wasn't present in earlier releases. Could you take a look to see if we should pull this in for 1.4?

@yhuai
Copy link
Contributor

yhuai commented May 20, 2015

I feel the problem is the json parser not the parquet writer. We should convert the string to UTF8Stirng during the parsing process. I will take a look.

@yhuai
Copy link
Contributor

yhuai commented May 20, 2015

https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/json/JacksonParser.scala#L150-160 looks suspicious. Also, https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/json/JacksonParser.scala#L179-188 looks like the same bug.

@davies
Copy link
Contributor

davies commented May 20, 2015

@yhuai @JoshRosen I will create another PR to fix these.

@davies davies mentioned this pull request May 20, 2015
Closed
@davies
Copy link
Contributor

davies commented May 20, 2015

This will be fixed by #6299

@yhuai
Copy link
Contributor

yhuai commented May 20, 2015

@viirya Thank you for the fix and test cases! However, seems the problem is the JSON parser part instead of the parquet writer side (see our above discussion). @davies has opened #6299 for the JSON parser part fix. How about we close this one?

@viirya
Copy link
Member Author

viirya commented May 20, 2015

Sure.

@viirya viirya closed this May 20, 2015
asfgit pushed a commit that referenced this pull request May 21, 2015
@yhuai
[SPARK-7565] [SQL] fix MapType in JsonRDD
3aa6185 
The key of Map in JsonRDD should be converted into UTF8String (also failed records), Thanks to yhuai viirya

Closes #6084

Author: Davies Liu <davies@databricks.com>

Closes #6299 from davies/string_in_json and squashes the following commits:

0dbf559 [Davies Liu] improve test, fix corrupt record
6836a80 [Davies Liu] move unit tests into Scala
b97af11 [Davies Liu] fix MapType in JsonRDD

(cherry picked from commit a25c1ab)
Signed-off-by: Yin Huai <yhuai@databricks.com>
asfgit pushed a commit that referenced this pull request May 21, 2015
@yhuai
[SPARK-7565] [SQL] fix MapType in JsonRDD
a25c1ab 
The key of Map in JsonRDD should be converted into UTF8String (also failed records), Thanks to yhuai viirya

Closes #6084

Author: Davies Liu <davies@databricks.com>

Closes #6299 from davies/string_in_json and squashes the following commits:

0dbf559 [Davies Liu] improve test, fix corrupt record
6836a80 [Davies Liu] move unit tests into Scala
b97af11 [Davies Liu] fix MapType in JsonRDD
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
@jeanlyn
[SPARK-7565] [SQL] fix MapType in JsonRDD
4b149bc 
The key of Map in JsonRDD should be converted into UTF8String (also failed records), Thanks to yhuai viirya

Closes apache#6084

Author: Davies Liu <davies@databricks.com>

Closes apache#6299 from davies/string_in_json and squashes the following commits:

0dbf559 [Davies Liu] improve test, fix corrupt record
6836a80 [Davies Liu] move unit tests into Scala
b97af11 [Davies Liu] fix MapType in JsonRDD
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
@jeanlyn
[SPARK-7565] [SQL] fix MapType in JsonRDD
bcb56ef 
The key of Map in JsonRDD should be converted into UTF8String (also failed records), Thanks to yhuai viirya

Closes apache#6084

Author: Davies Liu <davies@databricks.com>

Closes apache#6299 from davies/string_in_json and squashes the following commits:

0dbf559 [Davies Liu] improve test, fix corrupt record
6836a80 [Davies Liu] move unit tests into Scala
b97af11 [Davies Liu] fix MapType in JsonRDD
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@nemccarthy
[SPARK-7565] [SQL] fix MapType in JsonRDD
e9f646a 
The key of Map in JsonRDD should be converted into UTF8String (also failed records), Thanks to yhuai viirya

Closes apache#6084

Author: Davies Liu <davies@databricks.com>

Closes apache#6299 from davies/string_in_json and squashes the following commits:

0dbf559 [Davies Liu] improve test, fix corrupt record
6836a80 [Davies Liu] move unit tests into Scala
b97af11 [Davies Liu] fix MapType in JsonRDD
@viirya viirya deleted the fix_py_parquet_write branch December 27, 2023 18:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@viirya @SparkQA @JoshRosen @yhuai @davies