Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Support Y in from_unixtime #12216

Open
viadea opened this issue Feb 25, 2025 · 1 comment
Open

[FEA] Support Y in from_unixtime #12216

viadea opened this issue Feb 25, 2025 · 1 comment
Labels
? - Needs Triage Need team to review and classify cudf_dependency An issue or PR with this label depends on a new feature in cudf feature request New feature or request

Comments

@viadea
Copy link
Collaborator

viadea commented Feb 25, 2025

I wish we can support below fallback:

!Expression <FromUnixTime> from_unixtime(cast((cast(xxx as double) / 100.0) as bigint), Y-MM-dd, Some(Etc/UTC)) cannot run on GPU because Failed to convert Unsupported character: Y null

@viadea viadea added ? - Needs Triage Need team to review and classify feature request New feature or request labels Feb 25, 2025
@sameerz sameerz added the cudf_dependency An issue or PR with this label depends on a new feature in cudf label Feb 25, 2025
@viadea
Copy link
Collaborator Author

viadea commented Feb 26, 2025

Gen-AI can generate the code to repro above fallback message now:)

# Create a sample DataFrame with a timestamp column
data = [(1633072800000.0,)]
df = spark.createDataFrame(data, ["timestamp_ms"])
df.createOrReplaceTempView("sample_data")

spark.conf.set("spark.sql.legacy.timeParserPolicy","LEGACY")
# Perform the from_unixtime operation that causes the fallback
result = spark.sql("""
    SELECT 
        from_unixtime(
            CAST((CAST(timestamp_ms AS DOUBLE) / 100.0) AS BIGINT), 
            'Y-MM-dd'
        ) AS formatted_time 
    FROM sample_data
""")

# Show the result
result.show()
+--------------+
|formatted_time|
+--------------+
|    2487-07-02|
+--------------+

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
? - Needs Triage Need team to review and classify cudf_dependency An issue or PR with this label depends on a new feature in cudf feature request New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants