Metabase v0.48.4: Problem connecting CrateDB #153

amotl · 2024-11-21T15:41:25Z

Problem

On the comments to the video Metabase and CrateDB: Data Visualization, a few users, possibly @salamander101 and @JoeHan007, reported that:

It does not work with Metabase opensource 0.50.32.1 and CrateDB opensource 5.9.2.

amotl · 2024-11-21T15:50:09Z

Hi there,

thanks for your report. Can you provide more information around the problem you are running into, in order to support us to investigate and resolve this issue optimally? ¹

Maybe it stopped working with Metabase v0.50.32.1, released two weeks ago, while it worked before?
Maybe it was a short-term hiccup with Metabase, and has been resolved already?
Do you mind to try using Metabase v0.51.4.2, released today?
Did it stop working after upgrading to CrateDB 5.9.2 in the end, while it worked before?
What exactly means "It does not work"? Can you provide more details on the error(s) you are observing, and how to reproduce them?

With kind regards,
Andreas.

Thanks for providing version numbers from the very beginning. However, we need a better picture about the genesis, so we are humbly asking to read our questions, and/or report more extensively about what you think could be important for us to know, in order to resolve the problem. ↩

michaelkremmel · 2024-11-22T09:39:15Z

@amotl Thanks for creating this ticket. I tried this with the latest Metabase image from Docker, running locally and connecting it to a CRFREE Cluster in CrateDB Cloud. The issue I'm facing is that once I've created the database connection, it starts syncing tables, but it never stops. I've tried it with two different data samples, the weather_data first (9.3 MB), and since it took forever, I tried it with a smaller one, the marketing_data sample (217 KB), but the same here. I can click on "Dismiss sync spinner manually," and the spinner goes away, but the data is still not synced after ~15 minutes.

amotl · 2024-11-23T17:19:00Z

Hi again. We have been able to validate the integration with Metabase worked well until v0.48.3, and started failing with v0.48.4.

Metabase: Add test harness for validating Metabase against CrateDB cratedb-examples#750

It might be related to this SQL statement that reflects table privileges, which might have been updated, and trips CrateDB.

with table_privileges as (
 select
   NULL as role,
   t.schemaname as schema,
   t.objectname as table,
   pg_catalog.has_table_privilege(current_user, '"' || t.schemaname || '"' || '.' || '"' || t.objectname || '"',  'UPDATE') as update,
   pg_catalog.has_table_privilege(current_user, '"' || t.schemaname || '"' || '.' || '"' || t.objectname || '"',  'SELECT') as select,
   pg_catalog.has_table_privilege(current_user, '"' || t.schemaname || '"' || '.' || '"' || t.objectname || '"',  'INSERT') as insert,
   pg_catalog.has_table_privilege(current_user, '"' || t.schemaname || '"' || '.' || '"' || t.objectname || '"',  'DELETE') as delete
 from (
   select schemaname, tablename as objectname from pg_catalog.pg_tables
   union
   select schemaname, viewname as objectname from pg_catalog.pg_views
   union
   select schemaname, matviewname as objectname from pg_catalog.pg_matviews
 ) t
 where t.schemaname !~ '^pg_'
   and t.schemaname <> 'information_schema'
   and pg_catalog.has_schema_privilege(current_user, t.schemaname, 'USAGE')
)
select t.*
from table_privileges t;

SQLParseException[line 5:17: no viable alternative at input 'select\nNULL as role,\nt.schemaname as schema,\nt.objectname as table']

metabase  | 2024-11-23 17:06:13,684 WARN sync.util :: Error in sync step Sync postgres Database 2 'cratedb-testdrive'
metabase  | 2024-11-23 17:07:00,031 ERROR core.JobRunShell :: Job DEFAULT.metabase.task.sync-and-analyze.job threw an unhandled Exception:
metabase  | org.postgresql.util.PSQLException: ERROR: line 5:20: no viable alternative at input 'select\n   NULL as role,\n   t.schemaname as schema,\n   t.objectname as table'

With version v0.48.3, this was apparently just a warning, not failing the sync-and-analyze job.

metabase  | 2024-11-23 18:06:59,370 WARN sync.util :: Error running step 'sync-table-privileges' for postgres Database 2 'cratedb-testdrive'
metabase  | org.postgresql.util.PSQLException: ERROR: line 5:18: no viable alternative at input 'select\n  NULL as role,\n  t.schemaname as schema,\n  t.tablename as table'

amotl · 2024-11-23T17:56:51Z

[...] which might have been updated, and trips CrateDB.

In metabase/driver/postgres.clj, the difference between v0.48.3 and v0.48.4 is not significant, and the previous query raised the same error. What might be different now is that the outcome of the function driver/current-user-table-privileges is evaluated stronger, so the behavior is no longer graceful if that fails.

Maybe there are some hints in the Driver Interface Changelog for Metabase 0.49.0?

amotl · 2024-11-23T18:13:15Z

The reason why this SQL statement trips, by the way, is because it uses identifiers role, schema, and table, which are reserved in CrateDB.

When quoting them properly, this particular query works well.

with table_privileges as (
 select
   NULL as "role",
   t.schemaname as "schema",
   t.objectname as "table",
   pg_catalog.has_table_privilege(current_user, '"' || t.schemaname || '"' || '.' || '"' || t.objectname || '"',  'UPDATE') as "update",
   pg_catalog.has_table_privilege(current_user, '"' || t.schemaname || '"' || '.' || '"' || t.objectname || '"',  'SELECT') as "select",
   pg_catalog.has_table_privilege(current_user, '"' || t.schemaname || '"' || '.' || '"' || t.objectname || '"',  'INSERT') as "insert",
   pg_catalog.has_table_privilege(current_user, '"' || t.schemaname || '"' || '.' || '"' || t.objectname || '"',  'DELETE') as "delete"
 from (
   select schemaname, tablename as objectname from pg_catalog.pg_tables
   union
   select schemaname, viewname as objectname from pg_catalog.pg_views
   union
   select schemaname, matviewname as objectname from pg_catalog.pg_matviews
 ) t
 where t.schemaname !~ '^pg_'
   and t.schemaname <> 'information_schema'
   and pg_catalog.has_schema_privilege(current_user, t.schemaname, 'USAGE')
)
select t.*
from table_privileges t;

amotl · 2024-11-23T18:23:04Z

Proposals

For a start, we may try to improve the PostgreSQL driver on corresponding spots that would improve compatibility with CrateDB, and probe if upstream would accept relevant patches. First, we may want to try if improving this particular query yields some improvements when connecting CrateDB at all.

diff --git a/application/metabase/backlog.md b/application/metabase/backlog.md
index a1a5011..e00de4c 100644
--- a/application/metabase/backlog.md
+++ b/application/metabase/backlog.md
@@ -10,13 +10,13 @@ metabase  | org.postgresql.util.PSQLException: ERROR: line 5:20: no viable alter
 ```sql
 with table_privileges as (
  select
-   NULL as role,
-   t.schemaname as schema,
-   t.objectname as table,
-   pg_catalog.has_table_privilege(current_user, '"' || t.schemaname || '"' || '.' || '"' || t.objectname || '"',  'UPDATE') as update,
-   pg_catalog.has_table_privilege(current_user, '"' || t.schemaname || '"' || '.' || '"' || t.objectname || '"',  'SELECT') as select,
-   pg_catalog.has_table_privilege(current_user, '"' || t.schemaname || '"' || '.' || '"' || t.objectname || '"',  'INSERT') as insert,
-   pg_catalog.has_table_privilege(current_user, '"' || t.schemaname || '"' || '.' || '"' || t.objectname || '"',  'DELETE') as delete
+   NULL as "role",
+   t.schemaname as "schema",
+   t.objectname as "table",
+   pg_catalog.has_table_privilege(current_user, '"' || t.schemaname || '"' || '.' || '"' || t.objectname || '"',  'UPDATE') as "update",
+   pg_catalog.has_table_privilege(current_user, '"' || t.schemaname || '"' || '.' || '"' || t.objectname || '"',  'SELECT') as "select",
+   pg_catalog.has_table_privilege(current_user, '"' || t.schemaname || '"' || '.' || '"' || t.objectname || '"',  'INSERT') as "insert",
+   pg_catalog.has_table_privilege(current_user, '"' || t.schemaname || '"' || '.' || '"' || t.objectname || '"',  'DELETE') as "delete"
  from (
    select schemaname, tablename as objectname from pg_catalog.pg_tables
    union

If not, and maybe for the mid-term, my advise is to provide a dedicated CrateDB adapter to Metabase, deriving most of its functionality from the PostgreSQL adapter ¹, and only overriding specifics where needed. This would also give CrateDB a dedicated placement in Metabase's list of supported databases.

Possibly similar like the PostgreSQL <-> Redshift relationship and friends. ↩

amotl · 2024-12-05T03:34:19Z

After applying a fix,

[postgres] fix: Quote symbols in current-user-table-privileges metabase/metabase#50919

building Metabase,

docker build -t metabase:dev --build-arg MB_EDITION=oss --build-arg VERSION=v1.52.1.2-beta .

and using the testing rig to validate the improvement, the next error is revealed.

2024-12-05 03:30:20,514 ERROR sync.fetch-metadata :: Error while fetching metdata with 'db-metadata'
org.postgresql.util.PSQLException: ERROR: Unknown function: pg_catalog.has_any_column_privilege(CURRENT_USER, (((((('"' || replace(t.schemaname, '"', '""')) || '"') || '.') || '"') || replace(t.objectname, '"', '""')) || '"'), 'update')

Backlog: Create issue at crate/crate, following up on Implement pg_catalog.has_table_privilege crate#15748.

amotl · 2024-12-05T04:52:43Z

After removing sql-jdbc.sync/current-user-table-privileges :postgres, the next error is:

2024-12-05 04:50:36,987 ERROR sync.fetch-metadata :: Error while fetching metdata with 'db-metadata'
clojure.lang.ExceptionInfo: Error executing query: ERROR: Relation 'pg_stat_user_tables' unknown

amotl · 2024-12-05T12:15:11Z

After amending get-tables-sql in postgres.clj, to not use pg_stat_user_tables, those errors show up.
While syncing indexes might be secondary, syncing fields is essential. Otherwise, Metabase will bail out when accessing tables like this:

Error processing query: No fields found for table "summits".

Details

Origin: `sql-jdbc.sync/describe-fields-sql :postgres`

2024-12-05 10:25:26,222 WARN sync.util :: Error syncing Fields for Database ''postgres Database X ''CrateDB''''
clojure.lang.ExceptionInfo: Error executing query: ERROR: Cannot find data type: oid

 {:driver :postgres, :sql ["SELECT" "  \"c\".\"column_name\" AS \"name\"," "  CASE" "    WHEN \"c\".\"udt_schema\" IN ('public', 'pg_catalog') THEN FORMAT('%s', \"c\".\"udt_name\")" "    ELSE FORMAT('\"%s\".\"%s\"', \"c\".\"udt_schema\", \"c\".\"udt_name\")" "  END [....]

Schema Sync

2024-12-05 16:10:39,628 WARN sync.util :: Error running step ''sync-timezone'' for postgres Database 2 ''CrateDB''
org.postgresql.util.PSQLException: ERROR: Unknown session setting name 'timezone'.

2024-12-05 16:10:39,645 INFO sync.util :: STARTING: step ''sync-fields'' for postgres Database 2 ''CrateDB''
2024-12-05 16:10:39,679 WARN sync.describe-table :: Don't know how to map column type '_json' to a Field base_type, falling back to :type/*.
2024-12-05 16:10:39,679 WARN sync.describe-table :: Don't know how to map column type '_varchar' to a Field base_type, falling back to :type/*.
2024-12-05 16:10:39,680 WARN sync.describe-table :: Don't know how to map column type '_varchar' to a Field base_type, falling back to :type/*.

2024-12-05 16:10:39,755 INFO sync.describe-table :: Inferring schema for 1 JSON fields in Table 9 ''sys.allocations''
2024-12-05 16:10:39,760 ERROR sync-metadata.fields :: 
org.postgresql.util.PSQLException: ERROR: line 1:579: no viable alternative at input 'SELECT CASE WHEN 50000 < LENGTH(CAST("sys"."allocations"."decisions['explanations']" AS TEXT)) THEN NULL ELSE "sys"."allocations"."decisions['explanations']" END AS "decisions['explanations']" FROM "sys"."allocations" INNER JOIN ((SELECT "sys"."allocations"."table_schema", "sys"."allocations"."table_name", "sys"."allocations"."partition_ident", "sys"."allocations"."shard_id" FROM "sys"."allocations" ORDER BY "sys"."allocations"."table_schema" ASC, "sys"."allocations"."table_name" ASC, "sys"."allocations"."partition_ident" ASC, "sys"."allocations"."shard_id" ASC LIMIT $1) UNION'

2024-12-05 05:56:02,209 WARN sync.util :: Error syncing Indexes for Table X ''sys.XXX''
org.postgresql.util.PSQLException: ERROR: Unknown function: pg_catalog.pg_get_indexdef(tmp.ci_oid, tmp.ordinal_position, false)

2024-12-05 05:56:02,601 WARN sync.util :: Error running step ''sync-indexes'' for postgres Database X ''CrateDB''
clojure.lang.ExceptionInfo: class org.postgresql.util.PSQLException cannot be cast to class java.util.Map$Entry (org.postgresql.util.PSQLException is in unnamed module of loader 'app'; java.util.Map$Entry is in module java.base of loader 'bootstrap') {:toucan2/context-trace [["execute SQL with class com.mchange.v2.c3p0.impl.NewProxyConnection" {:toucan2.jdbc.query/sql-args ["SELECT * FROM \"METABASE_TABLE\" WHERE ((\"ACTIVE\" = TRUE) AND (\"VISIBILITY_TYPE\" IS NULL)) AND (\"DB_ID\" = ?)" 2]}]

2024-12-05 05:56:39,072 WARN api.common :: Unexpected parameters at [:post "/api/dataset"]: [:type :query :parameters]
Please add them to the schema or remove them from the API client
2024-12-05 05:56:39,242 ERROR query.permissions :: 
clojure.lang.ExceptionInfo: Error calculating permissions for query {:query {:database 2, :type :query, :query {:source-table 15}}}

2024-12-05 16:10:43,038 WARN sync.util :: Error fingerprinting Table 12 ''sys.cluster''
clojure.lang.ExceptionInfo: Error executing query: ERROR: line 2:83: no viable alternative at input 'SELECT ("sys"."cluster"."settings['cluster']['routing']['allocation']['balance']"#>'

amotl · 2024-12-05T16:57:38Z

The minimum changes needed to make CrateDB work again, using the most recent version of Metabase v0.52.1.1, are pretty minimal.

CrateDB: Subtract features from PostgreSQL adapter to make CrateDB work metabase#38

amotl mentioned this issue Nov 23, 2024

Metabase: Add test harness for validating Metabase against CrateDB crate/cratedb-examples#750

Merged

amotl changed the title ~~Metabase: Problem with Metabase 0.50.32.1 and CrateDB 5.9.2~~ Metabase: CrateDB stopped working with version 0.48.4 Nov 23, 2024

amotl added the bug Something isn't working label Nov 23, 2024

amotl changed the title ~~Metabase: CrateDB stopped working with version 0.48.4~~ Metabase v0.48.4: Problem connecting CrateDB Nov 23, 2024

amotl assigned kneth Nov 27, 2024

amotl mentioned this issue Dec 5, 2024

[postgres] fix: Quote symbols in current-user-table-privileges metabase/metabase#50919

Closed

1 task

amotl mentioned this issue Dec 5, 2024

Metabase: Update to v0.48.4 crate/cratedb-examples#761

Draft

amotl mentioned this issue Dec 5, 2024

CrateDB: Subtract features from PostgreSQL adapter to make CrateDB work crate/metabase#38

Draft

amotl mentioned this issue Dec 5, 2024

Metabase: Update to v0.52.1.1-cratedb crate/cratedb-examples#763

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Metabase v0.48.4: Problem connecting CrateDB #153

Metabase v0.48.4: Problem connecting CrateDB #153

amotl commented Nov 21, 2024

amotl commented Nov 21, 2024 •

edited

Loading

michaelkremmel commented Nov 22, 2024

amotl commented Nov 23, 2024 •

edited

Loading

amotl commented Nov 23, 2024 •

edited

Loading

amotl commented Nov 23, 2024 •

edited

Loading

amotl commented Nov 23, 2024 •

edited

Loading

amotl commented Dec 5, 2024 •

edited

Loading

amotl commented Dec 5, 2024

amotl commented Dec 5, 2024 •

edited

Loading

amotl commented Dec 5, 2024 •

edited

Loading

Metabase v0.48.4: Problem connecting CrateDB #153

Metabase v0.48.4: Problem connecting CrateDB #153

Comments

amotl commented Nov 21, 2024

Problem

amotl commented Nov 21, 2024 • edited Loading

Footnotes

michaelkremmel commented Nov 22, 2024

amotl commented Nov 23, 2024 • edited Loading

amotl commented Nov 23, 2024 • edited Loading

amotl commented Nov 23, 2024 • edited Loading

amotl commented Nov 23, 2024 • edited Loading

Proposals

Footnotes

amotl commented Dec 5, 2024 • edited Loading

amotl commented Dec 5, 2024

amotl commented Dec 5, 2024 • edited Loading

Details

Origin: sql-jdbc.sync/describe-fields-sql :postgres

Schema Sync

amotl commented Dec 5, 2024 • edited Loading

amotl commented Nov 21, 2024 •

edited

Loading

amotl commented Nov 23, 2024 •

edited

Loading

amotl commented Nov 23, 2024 •

edited

Loading

amotl commented Nov 23, 2024 •

edited

Loading

amotl commented Nov 23, 2024 •

edited

Loading

amotl commented Dec 5, 2024 •

edited

Loading

amotl commented Dec 5, 2024 •

edited

Loading

Origin: `sql-jdbc.sync/describe-fields-sql :postgres`

amotl commented Dec 5, 2024 •

edited

Loading