[SPARK-30579][DOC] Document ORDER BY Clause of SELECT statement in SQL Reference #27288

dilipbiswal · 2020-01-20T07:52:45Z

What changes were proposed in this pull request?

Document ORDER BY clause of SELECT statement in SQL Reference Guide.

Why are the changes needed?

Currently Spark lacks documentation on the supported SQL constructs causing
confusion among users who sometimes have to look at the code to understand the
usage. This is aimed at addressing this issue.

Does this PR introduce any user-facing change?

Yes.

Before:
There was no documentation for this.

After.

How was this patch tested?

Tested using jykyll build --serve

…erence

SparkQA · 2020-01-20T08:05:02Z

Test build #117094 has finished for PR 27288 at commit 581a1e9.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

dilipbiswal · 2020-01-20T08:26:32Z

retest this please

SparkQA · 2020-01-20T08:43:54Z

Test build #117101 has finished for PR 27288 at commit 581a1e9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-01-20T22:24:03Z

Test build #117137 has finished for PR 27288 at commit cb78d8d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

huaxingao · 2020-01-21T00:56:57Z

docs/sql-ref-syntax-qry-select-orderby.md

@@ -18,5 +18,121 @@ license: |
  See the License for the specific language governing permissions and
  limitations under the License.
 ---
+The <code>ORDER BY</code> clause is used to return the result rows in a sorted manner
+in the user specified order. Unlike the <code>SORT BY</code> clause, this clause guarantees


link to SORT BY?

@huaxingao will do it in the finalization pr.

OK if you want to take care of that at the end

huaxingao · 2020-01-21T00:57:23Z

docs/sql-ref-syntax-qry-select-orderby.md

+<dl>
+  <dt><code><em>ORDER BY</em></code></dt>
+  <dd>
+    Specifies a comma separated list of expressions along with optional parameters <code>sort_direction</code>


comma separated -> comma-separated?
I caught this because I saw Sean's comment in another PR :)

Actually in many places we have comma separated, for example,

An optional parameter that specifies a comma separated list of key and value pairs for partitions.

Not sure if I should open a MINOR PR to fix all of them.

huaxingao · 2020-01-21T01:07:05Z

I happen to see this:

Could you please fix this when you update the menu-sql.yaml?

dilipbiswal · 2020-01-21T03:14:45Z

Could you please fix this when you update the menu-sql.yaml?

Sure.. will do.

SparkQA · 2020-01-21T08:05:02Z

Test build #117156 has finished for PR 27288 at commit d1fb7bd.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2020-01-22T14:55:58Z

docs/sql-ref-syntax-qry-select-orderby.md

@@ -18,5 +18,121 @@ license: |
  See the License for the specific language governing permissions and
  limitations under the License.
 ---
+The <code>ORDER BY</code> clause is used to return the result rows in a sorted manner
+in the user specified order. Unlike the <code>SORT BY</code> clause, this clause guarantees


OK if you want to take care of that at the end

srowen · 2020-01-22T14:56:08Z

docs/sql-ref-syntax-qry-select-orderby.md

+    Optionally specifies whether to sort the rows in ascending (lowest to highest) or descending
+    (highest to lowest) order. The valid values for sort direction are <code>ASC</code> for ascending
+    and <code>DESC</code> for descending. If sort direction is not explicitly specified then by default
+    rows are sorted in ascending manner. <br><br>


just "sorted ascending"

srowen · 2020-01-22T14:57:42Z

docs/sql-ref-syntax-qry-select-orderby.md

+    sort direction. In Spark, NULL values are considered to be lower than any non-NULL values by default.
+    Therefore the ordering of NULL values depend on the sort direction.<br><br>
+    <ol>
+      <li>If the sort order is ASC, NULLS are returned first; to force NULLS to be last, use NULLS LAST</li>


I might turn this around, to emphasize the null sort order: "If NULLS FIRST (the default), then NULLs are returned first if sort order is ASC, and last if sort order is DESC" and likewise for the second items.

@srowen Hmmn.. trying to think if we are conveying properly with the above re-wording. So

If if NULLS FIRST is explicitly specified, you would ALWAYS see NULLs at the top of your resultset no matter what the sort order is.

if NULLS LAST is explicitly specified , you would ALWAYS see NULLS at the end no matter what the sort order is

.What do you think ?

Oh, my mistake, is that how it works? so NULLS FIRST puts nulls first no matter what the sort order? Maybe we can take the focus off of the sort order then. For example NULLS LAST does the same thing no matter what the sort order.

Maybe like: "NULLS FIRST forces NULLs to sort before all non-NULL values, regardless of the sort order" and likewise for NULLS LAST, and then explain that, if not specified, NULLs sort first if ASC and last if DESC.

@srowen Got it.. Will make the change.

SparkQA · 2020-01-22T23:50:08Z

Test build #117263 has finished for PR 27288 at commit a9dfbb4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2020-01-23T04:01:13Z

docs/sql-ref-syntax-qry-select-orderby.md

@@ -18,5 +18,125 @@ license: |
  See the License for the specific language governing permissions and
  limitations under the License.
 ---
+The <code>ORDER BY</code> clause is used to return the result rows in a sorted manner
+in the user specified order. Unlike the <code>SORT BY</code> clause, this clause guarantees
+total order in the output. 


total order -> a total order?

maropu · 2020-01-23T04:12:52Z

docs/sql-ref-syntax-qry-select-orderby.md

+  </dd>
+  <dt><code><em>sort_direction</em></code></dt>
+  <dd>
+    Optionally specifies whether to sort the rows in ascending (lowest to highest) or descending


we need the statement (lowest to highest)? ascending or descending is enough?

maropu · 2020-01-23T04:16:19Z

docs/sql-ref-syntax-qry-select-orderby.md

+  <dt><code><em>sort_direction</em></code></dt>
+  <dd>
+    Optionally specifies whether to sort the rows in ascending (lowest to highest) or descending
+    (highest to lowest) order. The valid values for sort direction are <code>ASC</code> for ascending


sort direction -> the sort direction?

SparkQA · 2020-01-23T06:10:04Z

Test build #117280 has finished for PR 27288 at commit ff61fa8.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-01-23T09:01:19Z

Test build #117292 has finished for PR 27288 at commit 7e46a5f.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-01-23T10:02:27Z

Test build #117297 has finished for PR 27288 at commit c409609.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen

@maropu are you OK with this? looks OK to me. If so we can move on to ORDER BY.

maropu

Yeah, it looks fine. Thanks, @dilipbiswal !

maropu · 2020-01-26T00:12:07Z

Thanks! Merged to master.

maropu · 2020-01-26T00:12:58Z

@dilipbiswal ok, lets move on to SORT BY.

dilipbiswal · 2020-01-26T00:23:24Z

Thanks a lot @srowen @huaxingao @maropu

@maropu Sure.. Let me double check on SORT by.

maropu · 2020-01-26T00:24:02Z

Thanks alot!

[SPARK-30579] Document ORDER BY Clause of SELECT statement in SQL Ref…

581a1e9

…erence

Code review

cb78d8d

huaxingao reviewed Jan 21, 2020

View reviewed changes

code review

d1fb7bd

srowen reviewed Jan 22, 2020

View reviewed changes

Code review

a9dfbb4

maropu reviewed Jan 23, 2020

View reviewed changes

Code review

ff61fa8

review

7e46a5f

minor

c409609

srowen reviewed Jan 25, 2020

View reviewed changes

maropu approved these changes Jan 26, 2020

View reviewed changes

maropu closed this in d5b92b2 Jan 26, 2020

dongjoon-hyun added the SQL label Feb 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-30579][DOC] Document ORDER BY Clause of SELECT statement in SQL Reference #27288

[SPARK-30579][DOC] Document ORDER BY Clause of SELECT statement in SQL Reference #27288

dilipbiswal commented Jan 20, 2020

SparkQA commented Jan 20, 2020

dilipbiswal commented Jan 20, 2020

SparkQA commented Jan 20, 2020

SparkQA commented Jan 20, 2020

huaxingao Jan 21, 2020

dilipbiswal Jan 21, 2020

srowen Jan 22, 2020

huaxingao Jan 21, 2020

huaxingao commented Jan 21, 2020

dilipbiswal commented Jan 21, 2020

SparkQA commented Jan 21, 2020

srowen Jan 22, 2020

srowen Jan 22, 2020

srowen Jan 22, 2020

dilipbiswal Jan 22, 2020 •

edited

Loading

srowen Jan 22, 2020

dilipbiswal Jan 22, 2020

SparkQA commented Jan 22, 2020

maropu Jan 23, 2020

maropu Jan 23, 2020

maropu Jan 23, 2020

SparkQA commented Jan 23, 2020

SparkQA commented Jan 23, 2020

SparkQA commented Jan 23, 2020

srowen left a comment

maropu left a comment

maropu commented Jan 26, 2020

maropu commented Jan 26, 2020

dilipbiswal commented Jan 26, 2020

maropu commented Jan 26, 2020

[SPARK-30579][DOC] Document ORDER BY Clause of SELECT statement in SQL Reference #27288

[SPARK-30579][DOC] Document ORDER BY Clause of SELECT statement in SQL Reference #27288

Conversation

dilipbiswal commented Jan 20, 2020

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Jan 20, 2020

dilipbiswal commented Jan 20, 2020

SparkQA commented Jan 20, 2020

SparkQA commented Jan 20, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

huaxingao commented Jan 21, 2020

dilipbiswal commented Jan 21, 2020

SparkQA commented Jan 21, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dilipbiswal Jan 22, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jan 22, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Jan 23, 2020

SparkQA commented Jan 23, 2020

SparkQA commented Jan 23, 2020

srowen left a comment

Choose a reason for hiding this comment

maropu left a comment

Choose a reason for hiding this comment

maropu commented Jan 26, 2020

maropu commented Jan 26, 2020

dilipbiswal commented Jan 26, 2020

maropu commented Jan 26, 2020

dilipbiswal Jan 22, 2020 •

edited

Loading