Set filter_and_project_min_output_page_row_count to 1 for final DistinctLimit

[kaikalur]

Addressing issue:

#17631

For specific case of DistinctLimit

Test plan - tests already exist

== RELEASE NOTES ==
General Changes
* Added a new optimization for showing results for (interactive) distinct limit N as they become available with no buffering.  This can be enabled using the session param: quick_distinct_limit_enabled to true.

[rschlussel]

what's the effect on queries that have lots of distinct values (didn't end up waiting a long time for results before)? Also, if the distinct limit isn't at the end of the query or it writes to a table, so the project isn't flushing straight to output.

[kaikalur]

Should be fine still because it's final step and the limit is generally low. Those are corner cases and they have other issues.

As for lot of distinct values, our RR shuffle fixes that part. Note that we are doing this for the FINAL distinct limit.

[kaikalur]

what's the effect on queries that have lots of distinct values (didn't end up waiting a long time for results before)? Also, if the distinct limit isn't at the end of the query or it writes to a table, so the project isn't flushing straight to output.

Also the issue we had with was not so much about having too many distinct values - it's about the limit - when the limit is greater than the number of distinct values, it was getting stuck. When the limit is lower than the distinct values, it's fast and that behavior hasn't changed as we simply short circuit the distinct limit.

[kaikalur]

Ping - anything else needed?

[rschlussel]

what's the effect on queries that have lots of distinct values (didn't end up waiting a long time for results before)? Also, if the distinct limit isn't at the end of the query or it writes to a table, so the project isn't flushing straight to output.

Also the issue we had with was not so much about having too many distinct values - it's about the limit - when the limit is greater than the number of distinct values, it was getting stuck. When the limit is lower than the distinct values, it's fast and that behavior hasn't changed as we simply short circuit the distinct limit.

Yeah, i understand how this fixes the limit issue. My concern is when there are a lot of distinct values (where it would have worked fine before) you can now end up creating lots of pages. Can we gate this with the distinct limit session property in case things go wrong?

[kaikalur]

what's the effect on queries that have lots of distinct values (didn't end up waiting a long time for results before)? Also, if the distinct limit isn't at the end of the query or it writes to a table, so the project isn't flushing straight to output.

Also the issue we had with was not so much about having too many distinct values - it's about the limit - when the limit is greater than the number of distinct values, it was getting stuck. When the limit is lower than the distinct values, it's fast and that behavior hasn't changed as we simply short circuit the distinct limit.

Yeah, i understand how this fixes the limit issue. My concern is when there are a lot of distinct values (where it would have worked fine before) you can now end up creating lots of pages. Can we gate this with the distinct limit session property in case things go wrong?

Note that this is completely indepedent of the total number of distinct values. It will have at most N pages if the LIMIT is N as we do this for only the FINAL distinctlimit. But sure I can rename and use the previous flag.

[kaikalur]

what's the effect on queries that have lots of distinct values (didn't end up waiting a long time for results before)? Also, if the distinct limit isn't at the end of the query or it writes to a table, so the project isn't flushing straight to output.

Also the issue we had with was not so much about having too many distinct values - it's about the limit - when the limit is greater than the number of distinct values, it was getting stuck. When the limit is lower than the distinct values, it's fast and that behavior hasn't changed as we simply short circuit the distinct limit.

Yeah, i understand how this fixes the limit issue. My concern is when there are a lot of distinct values (where it would have worked fine before) you can now end up creating lots of pages. Can we gate this with the distinct limit session property in case things go wrong?

Note that this is completely indepedent of the total number of distinct values. It will have at most N pages if the LIMIT is N as we do this for only the FINAL distinctlimit. But sure I can rename and use the previous flag.

Done.

I renamed the previous flag we were using for the RR shuffle as quick_distinct_limit_enabled so we can do these two together. Please take a look.