Spark Example

[michael-berk]

I'm leveraging spark to download overture data from S3. Sedona is incompatible with recent spark versions.

1. Would you take a spark Example like the one below?
2. Would you deprecate the sedona page in favor of the base spark version?

Psuedocode

import pyspark.sql.functions as F
from pyspark.sql import SparkSession

spark = SparkSession.builder.getOrCreate()

OVERTURE_RELEASE = "2025-01-22.0"
COUNTRY_CODES_OF_INTEREST = ["US", "GH"]
SOURCE_DATA_URL = f"s3a://overturemaps-us-west-2/release/{OVERTURE_RELEASE}/theme=places/type=place"
TARGET_TABLE = "my_catalog.my_schema.my_table"

country_overlap_condition = F.arrays_overlap(
    F.col("addresses.country"),
    F.array(*[F.lit(x.upper()) for x in COUNTRY_CODES_OF_INTEREST]),
)

source_df = (
    spark.read.format("parquet")
    .load(SOURCE_DATA_URL)
    .filter(country_overlap_condition)
    .withColumn("_overture_release_version", F.lit(OVERTURE_RELEASE))
    .withColumn("_ingest_timestamp", F.current_timestamp())
)

source_df.write.mode("append").format("delta").saveAsTable(TARGET_TABLE)

[danabauer]

Yes! You are welcome to submit a Spark example. Thank you. You're right, we definitely need to update that Sedona page.