Add support for save_table(..., mode="overwrite") to StatementExecutionBackend

[william-conti]

No description provided.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 84.58%. Comparing base (8921e0f) to head (77fe395).

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #74      +/-   ##
==========================================
+ Coverage   82.95%   84.58%   +1.62%     
==========================================
  Files           7        7              
  Lines         534      532       -2     
  Branches      105      105              
==========================================
+ Hits          443      450       +7     
+ Misses         55       50       -5     
+ Partials       36       32       -4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

❌ 17/18 passed, 1 failed, 2 skipped, 11m29s total

❌ test_overwrite: databricks.sdk.errors.platform.BadRequest: [INSUFFICIENT_PERMISSIONS] Insufficient privileges: (2.587s)

databricks.sdk.errors.platform.BadRequest: [INSUFFICIENT_PERMISSIONS] Insufficient privileges:
User does not have permission CREATE,USAGE on database `default`.
11:01 DEBUG [databricks.sdk] Loaded from environment
11:01 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred
11:01 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred
11:01 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service
11:01 INFO [databricks.sdk] Using Databricks Metadata Service authentication
[gw9] linux -- Python 3.10.13 /home/runner/work/lsql/lsql/.venv/bin/python
11:01 DEBUG [databricks.sdk] Loaded from environment
11:01 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred
11:01 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred
11:01 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service
11:01 INFO [databricks.sdk] Using Databricks Metadata Service authentication
11:01 DEBUG [databricks.labs.lsql.backends] [api][execute] CREATE TABLE IF NOT EXISTS default.foo (first STRING NOT NULL, second BOOLEAN NOT NULL) USING DE... (3 more bytes)
11:01 DEBUG [databricks.labs.lsql.core] Executing SQL statement: CREATE TABLE IF NOT EXISTS default.foo (first STRING NOT NULL, second BOOLEAN NOT NULL) USING DELTA
11:01 DEBUG [databricks.sdk] POST /api/2.0/sql/statements/
> {
>   "format": "JSON_ARRAY",
>   "statement": "CREATE TABLE IF NOT EXISTS default.foo (first STRING NOT NULL, second BOOLEAN NOT NULL) USING DE... (3 more bytes)",
>   "warehouse_id": "TEST_DEFAULT_WAREHOUSE_ID"
> }
< 200 OK
< {
<   "statement_id": "01eeec29-68e4-1703-9a59-272346f4700f",
<   "status": {
<     "error": {
<       "error_code": "BAD_REQUEST",
<       "message": "[INSUFFICIENT_PERMISSIONS] Insufficient privileges:\nUser does not have permission CREATE,USAGE o... (21 more bytes)"
<     },
<     "state": "FAILED"
<   }
< }
11:01 DEBUG [databricks.sdk] Loaded from environment
11:01 DEBUG [databricks.sdk] Ignoring pat auth, because metadata-service is preferred
11:01 DEBUG [databricks.sdk] Ignoring basic auth, because metadata-service is preferred
11:01 DEBUG [databricks.sdk] Attempting to configure auth: metadata-service
11:01 INFO [databricks.sdk] Using Databricks Metadata Service authentication
11:01 DEBUG [databricks.labs.lsql.backends] [api][execute] CREATE TABLE IF NOT EXISTS default.foo (first STRING NOT NULL, second BOOLEAN NOT NULL) USING DE... (3 more bytes)
11:01 DEBUG [databricks.labs.lsql.core] Executing SQL statement: CREATE TABLE IF NOT EXISTS default.foo (first STRING NOT NULL, second BOOLEAN NOT NULL) USING DELTA
11:01 DEBUG [databricks.sdk] POST /api/2.0/sql/statements/
> {
>   "format": "JSON_ARRAY",
>   "statement": "CREATE TABLE IF NOT EXISTS default.foo (first STRING NOT NULL, second BOOLEAN NOT NULL) USING DE... (3 more bytes)",
>   "warehouse_id": "TEST_DEFAULT_WAREHOUSE_ID"
> }
< 200 OK
< {
<   "statement_id": "01eeec29-68e4-1703-9a59-272346f4700f",
<   "status": {
<     "error": {
<       "error_code": "BAD_REQUEST",
<       "message": "[INSUFFICIENT_PERMISSIONS] Insufficient privileges:\nUser does not have permission CREATE,USAGE o... (21 more bytes)"
<     },
<     "state": "FAILED"
<   }
< }
[gw9] linux -- Python 3.10.13 /home/runner/work/lsql/lsql/.venv/bin/python

_{Running from acceptance #46}

[nfx]

let's take the risk

[nfx]

we'd need to properly support overwrites in the raw sql shape to keep the same'ish semantics as Spark:

1. INSERT INTO {full_name}_tmp ...
2. CREATE OR REPLACE TABLE {full_name} AS SELECT * FROM {full_name}_tmp
3. DROP TABLE {full_name}_tmp

otherwise the failure of overwrite will leave the table in a corrupt state.