Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-11967][SQL] Consistent use of varargs for multiple paths in DataFrameReader #9945

Closed
wants to merge 4 commits into from
Closed

[SPARK-11967][SQL] Consistent use of varargs for multiple paths in DataFrameReader #9945

wants to merge 4 commits into from

Conversation

rxin
Copy link
Contributor

@rxin rxin commented Nov 24, 2015

This patch makes it consistent to use varargs in all DataFrameReader methods, including Parquet, JSON, text, and the generic load function.

Also added a few more API tests for the Java API.

@rxin rxin changed the title [SPARK-11967][SQL] Use varargs for multiple paths in DataFrameReader [SPARK-11967][SQL] Consistent use of varargs for multiple paths in DataFrameReader Nov 24, 2015
@rxin
Copy link
Contributor Author

rxin commented Nov 24, 2015

cc @koertkuipers since you did the change for load(Array) - I'm changing it to varargs here.

@SparkQA
Copy link

SparkQA commented Nov 24, 2015

Test build #46623 has finished for PR 9945 at commit 01897a8.

  • This patch fails PySpark unit tests.
  • This patch does not merge cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 25, 2015

Test build #46638 has finished for PR 9945 at commit bd9a538.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):\n * public final class SortedIterator extends UnsafeSorterIterator\n * case class GetStructField(child: Expression, ordinal: Int, name: Option[String] = None)\n * case class PrettyAttribute(name: String, dataType: DataType = NullType)\n

@SparkQA
Copy link

SparkQA commented Nov 25, 2015

Test build #46641 has finished for PR 9945 at commit f97193c.

  • This patch fails PySpark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 25, 2015

Test build #46642 has finished for PR 9945 at commit c824f24.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46641/
Test FAILed.

asfgit pushed a commit that referenced this pull request Nov 25, 2015
@rxin
[SPARK-11967][SQL] Consistent use of varargs for multiple paths in Da…
4464fa2 
…taFrameReader

This patch makes it consistent to use varargs in all DataFrameReader methods, including Parquet, JSON, text, and the generic load function.

Also added a few more API tests for the Java API.

Author: Reynold Xin <rxin@databricks.com>

Closes #9945 from rxin/SPARK-11967.

(cherry picked from commit 25bbd3c)
Signed-off-by: Reynold Xin <rxin@databricks.com>
@asfgit asfgit closed this in 25bbd3c Nov 25, 2015
cloud-fan
cloud-fan reviewed Jan 27, 2016
View reviewed changes
sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java
@@ -298,4 +298,27 @@ public void pivot() {
Assert.assertEquals(48000.0, actual[1].getDouble(1), 0.01);
Assert.assertEquals(30000.0, actual[1].getDouble(2), 0.01);
}

public void testGenericLoad() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no @Test here, is it by means?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh we should add it. can you add it in ur next pr?

@HyukjinKwon HyukjinKwon mentioned this pull request May 1, 2016
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@rxin @SparkQA @AmplabJenkins @cloud-fan