Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for Ion data type #836

Closed
r0b0ji opened this issue Jun 20, 2021 · 2 comments
Closed

Support for Ion data type #836

r0b0ji opened this issue Jun 20, 2021 · 2 comments

Comments

@r0b0ji
Copy link

r0b0ji commented Jun 20, 2021

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
I want to use kryo serialization for for spark project where my data is in Ion format.

Describe alternatives you've considered
Currently when we encounter Ion data type in our spark job, we convert that to string to serialize and then again back from string to Ion format. This is wasteful as Ion binary compresses well and to be able to work natively with the data format will simplify the workflow.

Additional context
Ion is a superset of Json data type developed by Amazon. It works well for compression and has support for some more data type than Json. There are bindings available for most popular languages.

Wiki: https://en.wikipedia.org/wiki/Ion_(serialization_format)
Specs: https://amzn.github.io/ion-docs/
Ion-java github: https://github.com/amzn/ion-java

@r0b0ji
Copy link
Author

r0b0ji commented Jun 20, 2021

I have an implementation ready and willing to contribute. I contributed to magro/kryo-serializers repo some 2 months ago, but that repository seems to be inactive.

The pull request which I made in that repo: magro/kryo-serializers#130

@theigl
Copy link
Collaborator

theigl commented Jun 20, 2021

@r0b0ji: Kryo tries to avoid all third party dependencies so kryo-serializers is the right project for this.

@magro: Could you take a look at this PR?

@theigl theigl closed this as completed Apr 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

2 participants