feat: Add Apache Doris support #24714

liujiwen-up · 2023-07-17T08:53:35Z

SUMMARY

The Apache Doris database is based on the MySQL protocol and introduces the concept of Multi Catalog. Therefore, the engine specification of mysqldb will be incompatible with the catalog, and some field types are not the same as those in mysql. To fix these issues, I added the Apache Doris engine specification.

Main difference:

Supports the connection of Doris's Multi Catalog
Fixed the problem that only "BASE TABLE" was displayed in information_schema
Fixed data types for JSONB, DATEV2, DATETIMEV2, DECIMALV3, MAP, STRUCT, LARGEINT, QUANTILE_STATE and AGG_STATE

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

Has associated issue:
Required feature flags:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

github-actions

Congrats on making your first PR and thank you for contributing to Superset! 🎉 ❤️

We hope to see you in our Slack community too! Not signed up? Use our Slack App to self-register.

codecov · 2023-07-17T13:08:32Z

Codecov Report

Attention: 8 lines in your changes are missing coverage. Please review.

Comparison is base (07551dc) 69.06% compared to head (1eaeaef) 69.09%.

Files	Patch %	Lines
superset/db_engine_specs/doris.py	89.18%	8 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #24714      +/-   ##
==========================================
+ Coverage   69.06%   69.09%   +0.03%     
==========================================
  Files        1939     1940       +1     
  Lines       75796    75870      +74     
  Branches     8444     8444              
==========================================
+ Hits        52349    52426      +77     
+ Misses      21271    21268       -3     
  Partials     2176     2176

Flag	Coverage Δ
hive	`53.68% <79.72%> (+0.05%)`	⬆️
javascript	`56.27% <ø> (ø)`
mysql	`78.16% <79.72%> (?)`
postgres	`78.26% <79.72%> (+<0.01%)`	⬆️
presto	`53.64% <79.72%> (+0.05%)`	⬆️
python	`82.95% <89.18%> (+0.04%)`	⬆️
sqlite	`76.91% <79.72%> (+<0.01%)`	⬆️
unit	`55.79% <89.18%> (+0.06%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

liujiwen-up · 2023-09-25T02:14:10Z

LGTM

liujiwen-up · 2023-09-25T02:45:57Z

@villebro Hello, can you help me review it?

ThreatLEVELred

wow that was two hours straight

docs/docs/databases/doris.mdx

liujiwen-up · 2023-10-19T09:10:50Z

LGTM

liujiwen-up · 2023-10-19T09:35:49Z

@villebro
Can you help trigger workflows?

liujiwen-up · 2023-10-23T06:40:57Z

@betodealmeida Hello, excuse me for a few minutes, can you help me review it?

john-bodley · 2023-11-08T17:27:34Z

@liujiwen-up I've triggered the workflows for you.

john-bodley

Thanks for adding a new engine and all the associated tests.

@betodealmeida would you mind taking a look at this change as you seem to be the resident database expert.

john-bodley · 2023-11-08T23:17:25Z

docs/docs/databases/doris.mdx

+
+## Doris
+
+The [sqlalchemy-doris](https://pypi.org/project/pydoris/) library is the recommended


Suggested change

The [sqlalchemy-doris](https://pypi.org/project/pydoris/) library is the recommended

The [sqlalchemy-doris](https://pypi.org/project/pydoris/) library is the recommended way to connect to Apache Doris through SQLAlchemy.

john-bodley · 2023-11-08T23:17:33Z

docs/docs/databases/doris.mdx

+## Doris
+
+The [sqlalchemy-doris](https://pypi.org/project/pydoris/) library is the recommended
+way to connect to Apache Doris through SQLAlchemy.


Suggested change

way to connect to Apache Doris through SQLAlchemy.

john-bodley · 2023-11-08T23:20:17Z

setup.py

@@ -205,6 +205,7 @@ def get_git_sha() -> str:
        "vertica": ["sqlalchemy-vertica-python>=0.5.9, < 0.6"],
        "netezza": ["nzalchemy>=11.0.2"],
        "starrocks": ["starrocks>=1.0.0"],
+        "doris": ["pydoris>=1.0.0"],


Could we include an upper bound of 2.0.0 to prevent potenitial breaking changes when the dependencies are bumped?

OK I will modify it, thanks for your suggestion

superset/db_engine_specs/doris.py

rusackas · 2023-11-21T19:30:57Z

Thank you for adding this! We'll try to get it into Superset. 3.1 officially. Happy to continue supporting these integrations in the Apache ecosystem :)

Co-authored-by: Evan Rusackas <evan@preset.io>

liujiwen-up added 2 commits July 17, 2023 16:26

Add Apache Doris support

b080af9

Add Apache Doris support

090662e

pull-request-size bot added the size/L label Jul 17, 2023

github-actions bot reviewed Jul 17, 2023

View reviewed changes

john-bodley requested review from betodealmeida, john-bodley and villebro July 18, 2023 16:58

liujiwen-up changed the title ~~Add Apache Doris support~~ feat:Add Apache Doris support Sep 25, 2023

ThreatLEVELred reviewed Sep 25, 2023

View reviewed changes

docs/docs/databases/doris.mdx Show resolved Hide resolved

liujiwen-up requested a review from ThreatLEVELred September 25, 2023 15:44

Merge branch 'apache:master' into doris

8a5bd2d

liujiwen-up changed the title ~~feat:Add Apache Doris support~~ feat: Add Apache Doris support Oct 19, 2023

liujiwen-up added 2 commits October 23, 2023 14:36

Update installing-database-drivers.mdx

b2fbbe3

Merge branch 'apache:master' into doris

3039078

fix spelling mistake

40e1751

john-bodley reviewed Nov 8, 2023

View reviewed changes

liujiwen-up and others added 6 commits November 9, 2023 11:27

Merge branch 'apache:master' into doris

e2d8164

Merge remote-tracking branch 'origin/master'

14ff24f

Made some minor changes

43b6a3b

Merge remote-tracking branch 'origin/doris'

210e161

fix pylint error

8c94e25

Merge branch 'master' into doris

1eaeaef

rusackas added the v3.0 Label added by the release manager to track PRs to be included in the 3.0 branch label Nov 21, 2023

rusackas requested review from michael-s-molina and removed request for ThreatLEVELred November 21, 2023 19:29

rusackas approved these changes Nov 21, 2023

View reviewed changes

rusackas merged commit 9712146 into apache:master Nov 21, 2023
35 checks passed

michael-s-molina removed the v3.0 Label added by the release manager to track PRs to be included in the 3.0 branch label Dec 4, 2023

josedev-union pushed a commit to Ortege-xyz/studio that referenced this pull request Jan 22, 2024

feat: Add Apache Doris support (apache#24714)

83ee9c4

Co-authored-by: Evan Rusackas <evan@preset.io>

cccs-rc pushed a commit to CybercentreCanada/superset that referenced this pull request Mar 6, 2024

feat: Add Apache Doris support (apache#24714)

0927b78

Co-authored-by: Evan Rusackas <evan@preset.io>

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 3.1.0 labels Mar 8, 2024

sfirke pushed a commit to sfirke/superset that referenced this pull request Mar 22, 2024

feat: Add Apache Doris support (apache#24714)

52f7ac5

Co-authored-by: Evan Rusackas <evan@preset.io>

vinothkumar66 pushed a commit to vinothkumar66/superset that referenced this pull request Nov 11, 2024

feat: Add Apache Doris support (apache#24714)

fd33e84

Co-authored-by: Evan Rusackas <evan@preset.io>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add Apache Doris support #24714

feat: Add Apache Doris support #24714

liujiwen-up commented Jul 17, 2023 •

edited by john-bodley

Loading

github-actions bot left a comment

codecov bot commented Jul 17, 2023 •

edited

Loading

liujiwen-up commented Sep 25, 2023

liujiwen-up commented Sep 25, 2023

ThreatLEVELred left a comment

liujiwen-up commented Oct 19, 2023

liujiwen-up commented Oct 19, 2023

liujiwen-up commented Oct 23, 2023 •

edited

Loading

john-bodley commented Nov 8, 2023

john-bodley left a comment •

edited

Loading

john-bodley Nov 8, 2023

john-bodley Nov 8, 2023

john-bodley Nov 8, 2023

liujiwen-up Nov 9, 2023

rusackas commented Nov 21, 2023


		## Doris

		The [sqlalchemy-doris](https://pypi.org/project/pydoris/) library is the recommended

feat: Add Apache Doris support #24714

feat: Add Apache Doris support #24714

Conversation

liujiwen-up commented Jul 17, 2023 • edited by john-bodley Loading

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

github-actions bot left a comment

Choose a reason for hiding this comment

codecov bot commented Jul 17, 2023 • edited Loading

Codecov Report

liujiwen-up commented Sep 25, 2023

liujiwen-up commented Sep 25, 2023

ThreatLEVELred left a comment

Choose a reason for hiding this comment

liujiwen-up commented Oct 19, 2023

liujiwen-up commented Oct 19, 2023

liujiwen-up commented Oct 23, 2023 • edited Loading

john-bodley commented Nov 8, 2023

john-bodley left a comment • edited Loading

Choose a reason for hiding this comment

john-bodley Nov 8, 2023

Choose a reason for hiding this comment

john-bodley Nov 8, 2023

Choose a reason for hiding this comment

john-bodley Nov 8, 2023

Choose a reason for hiding this comment

liujiwen-up Nov 9, 2023

Choose a reason for hiding this comment

rusackas commented Nov 21, 2023

liujiwen-up commented Jul 17, 2023 •

edited by john-bodley

Loading

codecov bot commented Jul 17, 2023 •

edited

Loading

liujiwen-up commented Oct 23, 2023 •

edited

Loading

john-bodley left a comment •

edited

Loading