Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Scala ArrayBuffer cast to Spark ArrayData for Scala 2.13 #689

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

hipp0gryph
Copy link

Closes #688

Fix problem with cast scala.collection.mutable.ArrayBuffer to org.apache.spark.sql.catalyst.util.ArrayData

@srowen
Copy link
Collaborator

srowen commented Aug 11, 2024

Hm, I don't think it's quite that simple unfortunately - it causes some test to fail.
I'm not totally sure why. It seems like building an Array this way doesn't quite work, ends up with 1 element not 3 for some reason, when 3 should be the result of the array xs_any column.

It may be that you have to try returning an array later somewhere, I'm not sure.

We'd also have to make a change in spark to match this, as the code really lives there now, but that's for later.

@hipp0gryph
Copy link
Author

Hm, I don't think it's quite that simple unfortunately - it causes some test to fail. I'm not totally sure why. It seems like building an Array this way doesn't quite work, ends up with 1 element not 3 for some reason, when 3 should be the result of the array xs_any column.

It may be that you have to try returning an array later somewhere, I'm not sure.

We'd also have to make a change in spark to match this, as the code really lives there now, but that's for later.

Okay) I really bad in Scala. After that errors I know my target on future :) I think problem in immutability Array type. I think that problem need harder solution, but we can try start tests and dream about success)

@hipp0gryph
Copy link
Author

@srowen Hello! Can you run tests, please? Thank you in advance!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Problem use Scala functions in python. Scala error on Spark 3.3.2. Scala Version 2.13.8.
2 participants