Databricks tests

[zerodarkzone]

Hi,

I've added tests suport to the databricks backend and while doing it also fixed a couple of problems I've found.

[eakmanrq]

Really appreciate your contribution of these tests. 🙏

I made a few comments but can you also summarize some of the key differences you are seeing with the Databricks engine compared to the PySpark API? From what I can tell it was the following:

• Timestamp differences (I'm currently refactoring timestamp logic due to a bugfix in SQLGlot that may help. What you have currently is fine though)
• Differences in how arrays/maps are handled

Let me know if there is anything else worth noting.

In terms of the maps issue, is there a way that we can detect a map and have it return a dict? You can see an example with DuckDB where it checks the value in the response and tries to determine if it is a map and if so converts it to a dict:

sqlframe/sqlframe/duckdb/session.py

Lines 59 to 69 in ed4c24a

	@classmethod
	def _try_get_map(cls, value: t.Any) -> t.Optional[t.Dict[str, t.Any]]:
	if value and isinstance(value, dict):
	# DuckDB < 1.1.0 support
	if "key" in value and "value" in value:
	return dict(zip(value["key"], value["value"]))
	# DuckDB >= 1.1.0 support
	# If a key is not a string then it must not represent a column and therefore must be a map
	if len([k for k in value if not isinstance(k, str)]) > 0:
	return value
	return None

[zerodarkzone]

Really appreciate your contribution of these tests. 🙏

I made a few comments but can you also summarize some of the key differences you are seeing with the Databricks engine compared to the PySpark API? From what I can tell it was the following:

• Timestamp differences (I'm currently refactoring timestamp logic due to a bugfix in SQLGlot that may help. What you have currently is fine though)
• Differences in how arrays/maps are handled

Let me know if there is anything else worth noting.

In terms of the maps issue, is there a way that we can detect a map and have it return a dict? You can see an example with DuckDB where it checks the value in the response and tries to determine if it is a map and if so converts it to a dict:

sqlframe/sqlframe/duckdb/session.py

Lines 59 to 69 in ed4c24a

	@classmethod
	def _try_get_map(cls, value: t.Any) -> t.Optional[t.Dict[str, t.Any]]:
	if value and isinstance(value, dict):
	# DuckDB < 1.1.0 support
	if "key" in value and "value" in value:
	return dict(zip(value["key"], value["value"]))
	# DuckDB >= 1.1.0 support
	# If a key is not a string then it must not represent a column and therefore must be a map
	if len([k for k in value if not isinstance(k, str)]) > 0:
	return value
	return None

Hi, I'll give it a try

[zerodarkzone]

Hi,
I've found a problem with this function.

When you have a map, it will covert all the keys to lowercase so it makes this test fail:

[eakmanrq]

When you have a map, it will covert all the keys to lowercase so it makes this test fail:

Yeah I think I know why. Since upper is uppercasing the columns in a struct, and they aren't quoted, during normalization it seems the columns as case-insensitive (since they aren't quoted) and therefore the uppercase can become lower case.

This is a tricky case that is more of an edge case. Want to transform the keys in some other ways that can be measured and do an instance check if it is Databricks and do a special test just for Databricks? Maybe add a character or something.

[zerodarkzone]

When you have a map, it will covert all the keys to lowercase so it makes this test fail:

Yeah I think I know why. Since upper is uppercasing the columns in a struct, and they aren't quoted, during normalization it seems the columns as case-insensitive (since they aren't quoted) and therefore the uppercase can become lower case.

This is a tricky case that is more of an edge case. Want to transform the keys in some other ways that can be measured and do an instance check if it is Databricks and do a special test just for Databricks? Maybe add a character or something.

Hi,
This is done

[eakmanrq]

Thanks @zerodarkzone! Feel free to merge if you are ready.

[zerodarkzone]

Hi,
I think everything is ready to merge. I don't have permission to do it. Could You merge it?