You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
I want to use kryo serialization for for spark project where my data is in Ion format.
Describe alternatives you've considered
Currently when we encounter Ion data type in our spark job, we convert that to string to serialize and then again back from string to Ion format. This is wasteful as Ion binary compresses well and to be able to work natively with the data format will simplify the workflow.
Additional context
Ion is a superset of Json data type developed by Amazon. It works well for compression and has support for some more data type than Json. There are bindings available for most popular languages.
I have an implementation ready and willing to contribute. I contributed to magro/kryo-serializers repo some 2 months ago, but that repository seems to be inactive.
Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
I want to use kryo serialization for for spark project where my data is in Ion format.
Describe alternatives you've considered
Currently when we encounter Ion data type in our spark job, we convert that to string to serialize and then again back from string to Ion format. This is wasteful as Ion binary compresses well and to be able to work natively with the data format will simplify the workflow.
Additional context
Ion is a superset of Json data type developed by Amazon. It works well for compression and has support for some more data type than Json. There are bindings available for most popular languages.
Wiki: https://en.wikipedia.org/wiki/Ion_(serialization_format)
Specs: https://amzn.github.io/ion-docs/
Ion-java github: https://github.com/amzn/ion-java
The text was updated successfully, but these errors were encountered: