Introduce unified DataSourceExec for provided datasources, remove ParquetExec, CsvExec, etc

[mertak-synnada]

Which issue does this PR close?

• Closes [DISCUSS] Single Source ExecutionPlan Across All TableProviders #13838.
• Closes Support Limit pushdown for MemoryExec #14502
• Closes Implement fetch limit for MemoryExec #14337

Rationale for this change

This PR merges all Data sources into one Execution Plan, named DataSourceExec, and a single trait named DataSource which is inspired by DataSink. File-related shared behaviors are merged into FileSourceConfig, and a MemorySourceConfig is created. MemoryExec, ArrowExec, AvroExec, CsvExec, NdJsonExec, and ParquetExec are deprecated and merged into DataSourceExec.

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

[alamb]

Something else I noticed is the import paths for ParquetExec are gone (so I got a lot of compile errors about missing ParquetExec):

error[E0432]: unresolved import `datafusion::datasource::physical_plan::ParquetExec`
 --> iox_query/src/physical_optimizer/limits.rs:4:5
  |
4 | use datafusion::datasource::physical_plan::ParquetExec;
  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ no `ParquetExec` in `datasource::physical_plan`
  |
help: consider importing this struct instead
  |
4 | use datafusion::datasource::physical_plan::parquet::ParquetExec;
  |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Suggested action is reexport the file here

--- a/datafusion/core/src/datasource/physical_plan/mod.rs
+++ b/datafusion/core/src/datasource/physical_plan/mod.rs
@@ -33,7 +33,7 @@ pub(crate) use self::json::plan_to_json;
 #[cfg(feature = "parquet")]
 pub use self::parquet::source::ParquetSource;
 #[cfg(feature = "parquet")]
-pub use self::parquet::{ParquetFileMetrics, ParquetFileReaderFactory};
+pub use self::parquet::{ParquetFileMetrics, ParquetFileReaderFactory, ParquetExec};

[alamb]

Great! The change you suggested makes sense. Since it is a very small change, we'll shortly add it to this PR and so migration smoother to others as well.

To be clear, I think we can do these kinds of changes in a follow on PR too -- I haven't found any blockers to this PR being merged yet but I don't have our tests running yet to be sure.

But all in all things are going well from my perspective

[alamb]

I got enough of our code to compile and tests running that i think this PR is ok to merge. Thank you @mertak-synnada @ozankabak and @berkaysynnada -- this is pretty epic

I believe this will be a non trivial change for any system that directly creates and manipulates ParquetExec but I think the new structures seem to support everything we need.

It would be really great to write up a document (maybe a blog post) that explains this change and gives help for people upgrading. Specifically some examples of creating ParquetExec before and after (and something similar to CsvExec and AvroExec.

As I work through the rest of our upgrade, I may have additional ideas to make it easier.

I can't realistically review this PR in detail, bug I trust that other committers have done so 👍

[ozankabak]

I got enough of our code to compile and tests running that i think this PR is ok to merge. Thank you @mertak-synnada @ozankabak and @berkaysynnada -- this is pretty epic

Great, let's go ahead 🚀

It would be really great to write up a document (maybe a blog post) that explains this change and gives help for people upgrading. Specifically some examples of creating ParquetExec before and after (and something similar to CsvExec and AvroExec.

This is a terrific idea -- we will be happy to write up a blog post that will serve a dual purpose: (1) help people with upgrading, and (2) brag about this epic PR 🙂

As I work through the rest of our upgrade, I may have additional ideas to make it easier.

Great, we will be happy to collaborate on any follow-on PRs.

[ozankabak]

I just merged the PR -- I will check back in half an hour to see if there are any problems

[alamb]

This is a terrific idea -- we will be happy to write up a blog post that will serve a dual purpose: (1) help people with upgrading, and (2) brag about this epic PR 🙂

Yes 100% that would be perfect. I think such a post would be find to put on the datafusion blog but content marketing wise if you write it putting it on a synnada blog would make total sense too

[alamb]

Here is one small follow up from this PR:

• Improve UX Rename FileScanConfig::new_exec to FileScanConfig::build #14670

[alamb]

As I have been working through actually upgrading delta.rs with these changes it turns out it was more effort than expected (like everything in software)

I have created a few PRs that I think would be nice to get in before we release this change downstream

• Require Debug for DataSource #14882
• Include struct name on FileScanConfig debug impl #14883
• Rename DataSource and FileSource fields for consistency #14898

[alamb]

It would be really great to write up a document (maybe a blog post) that explains this change and gives help for people upgrading. Specifically some examples of creating ParquetExec before and after (and something similar to CsvExec and AvroExec.

Also I started writing down an upgrade guide (not yet ready for review)

• Add Upgrade Guide for DataFusion 46.0.0 #14891