Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add duckdb build/concepts and use SQLGlot to convert BigQuery SQL into other dialects #1689

Merged
merged 47 commits into from
Feb 20, 2024
Merged
Changes from 1 commit
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
463609d
First "working" version (all SQL runs except for a few pivot concepts…
Apr 26, 2023
b7fffcc
Stripped dead code
Apr 26, 2023
ecbfe4d
Simplify pathing, strip more dead code
Apr 26, 2023
0e02dad
Align with shell script version
Apr 26, 2023
10e7f4e
Add requirements.txt
Apr 26, 2023
17ac9f4
Pulled sql out of .sh plus minor related mods
Apr 27, 2023
f777f46
Add table creation/loading
Apr 27, 2023
dc53b7b
Schema support (option) to mirror psql version
Apr 27, 2023
c52d421
Another chunk of dead code
Apr 27, 2023
2d3ef83
Big bug in DatetimeDiff implementation!
Apr 28, 2023
a173fd0
Adding missing fluid_balance views
Apr 28, 2023
f10eeb4
missed deletion
Apr 28, 2023
3aefb39
Rename in line with Postgres
Apr 28, 2023
0645b6c
Adding indexes
Apr 28, 2023
06a7422
Added checks
Apr 28, 2023
212f8bd
Updating README.md
Apr 28, 2023
f7a9729
Missed file rename
Apr 29, 2023
d4b5426
Move fake CHARTEVENTS PK to indexes script -- this may fail on machin…
SphtKr May 1, 2023
6bdcf14
Fixed outright errors
May 2, 2023
71e8d54
Experimental option to use integer or fractional DATETIME_DIFF function
May 8, 2023
2854015
simplify parse to use DATETIME_TRUNC and use unique alias
alistairewj Nov 24, 2023
4166507
explicitly cast seconds to integer
alistairewj Nov 24, 2023
b918099
move duckdb concept file to new python package folder
alistairewj Nov 24, 2023
62f4f59
tidy up readme and update to v2.2
alistairewj Nov 24, 2023
c79355a
init python package to support converting SQL scripts across dialects
alistairewj Nov 24, 2023
ae7a4c0
add step to test mimic_utils
alistairewj Nov 24, 2023
24de255
remove unneeded duckdb concept files
alistairewj Nov 24, 2023
f2f9148
move sqlglot monkey patching to individual modules
alistairewj Nov 24, 2023
3b133f8
reorganize classes to top
alistairewj Nov 27, 2023
12bdf7e
init duckdb transforms
alistairewj Nov 27, 2023
0654e16
add mimic_utils module name to import
alistairewj Nov 27, 2023
084a15f
add duckdb to transpilation
alistairewj Nov 27, 2023
7d54f86
remove semi-colon
alistairewj Nov 27, 2023
dcb8cc0
explicitly cast upper limit of generate series as an integer
alistairewj Nov 27, 2023
eec813b
add derived schema name to scripts by default
alistairewj Nov 27, 2023
1bcceb0
rename subfolder as it contains sqlglot transformations
alistairewj Nov 27, 2023
1ff7f57
update import
alistairewj Nov 27, 2023
9d2d4dc
refactor mimic-iv SQL queries into subfolders of concepts
alistairewj Dec 18, 2023
24d177b
formatting fixes by sqlfluff
alistairewj Dec 18, 2023
7487bbe
update postgres concepts with new transpile method
alistairewj Dec 19, 2023
51c6c4f
add duckdb concepts for mimic-iv
alistairewj Dec 19, 2023
7bb12b8
update README for mimic-iv
alistairewj Dec 23, 2023
9b083be
add help text for the mimic_utils entry points
alistairewj Jan 6, 2024
f1f07b8
switch to hatchling backend and only include mimic_utils package
alistairewj Jan 6, 2024
ad08bae
readme for pypi package
alistairewj Jan 6, 2024
f826074
various typo fixes and clearer language
alistairewj Feb 20, 2024
1dfa41c
add setup python to workflow
alistairewj Feb 20, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Move fake CHARTEVENTS PK to indexes script -- this may fail on machin…
…es without a large amount of RAM, and this way it can be skipped with `--skip-indexes`
SphtKr authored and alistairewj committed Dec 18, 2023

Verified

This commit was signed with the committer’s verified signature.
alistairewj Alistair Johnson
commit d4b5426c5ec43a2ed7289fb300961b7b17fc54e3
8 changes: 8 additions & 0 deletions mimic-iii/buildmimic/duckdb/duckdb_add_indexes.sql
Original file line number Diff line number Diff line change
@@ -544,3 +544,11 @@ CREATE INDEX TRANSFERS_idx03
-- DROP INDEX IF EXISTS TRANSFERS_idx05;
-- CREATE INDEX TRANSFERS_idx05
-- ON TRANSFERS (LOS);

--------------------------------------
-- CHARTEVENTS PRIMARY KEY SUBSTITUTE
--------------------------------------

-- FIXME: Remove this index when the PK can be re-added...
CREATE UNIQUE INDEX chartevents_rowid_pk ON CHARTEVENTS (ROW_ID);

2 changes: 0 additions & 2 deletions mimic-iii/buildmimic/duckdb/duckdb_add_tables.sql
Original file line number Diff line number Diff line change
@@ -87,8 +87,6 @@ CREATE TABLE CHARTEVENTS
-- See https://github.com/duckdb/duckdb/issues/6668#issuecomment-1474880266
--,CONSTRAINT chartevents_rowid_pk PRIMARY KEY (ROW_ID)
);
-- Remove this index when the PK can be re-added...
CREATE UNIQUE INDEX chartevents_rowid_pk ON CHARTEVENTS (ROW_ID);

DROP TABLE IF EXISTS CPTEVENTS CASCADE;
CREATE TABLE CPTEVENTS