Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework on serialization and deserialization #252

Closed
wants to merge 9 commits into from

Conversation

Yury-Fridlyand
Copy link

@Yury-Fridlyand Yury-Fridlyand commented Mar 21, 2023

Description

Rework PhysicalPlan tree serialization and deserialization. Make it scalable and extendable.
If a new plan should be serialized (a new feature is becoming supported by pagination) - it should properly overload SerializablePlan::writeExternal only.
The plan tree will be recovered in deserialization the same way it was before serialization.
New mechanism can also skip plans from serialization - for example, it skips ResourceMonitorPlan.

Issues Resolved

opensearch-project#1483

Check List

  • New functionality includes testing.
    • All tests pass, including unit test, integration test and doctest
  • New functionality has been documented.
    • New functionality has javadoc added
    • New functionality has user manual doc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>

Step 2.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
@Yury-Fridlyand Yury-Fridlyand changed the base branch from dev-pagination-v2_revC to integ-pagination-p1 March 29, 2023 20:33
@Yury-Fridlyand Yury-Fridlyand force-pushed the dev-pagination-v2_revD branch from 7a9d285 to ef44600 Compare March 29, 2023 20:33
Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
@codecov

This comment was marked as spam.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
return new Cursor(CURSOR_PREFIX
+ serialize(((SerializablePlan) plan).getPlanForSerialization()));
// ClassCastException thrown when a plan in the tree doesn't implement SerializablePlan
} catch (ClassCastException | NoCursorException e) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When can ClassCastException be thrown?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are always trying to convert a plan tree to a cursor in


If tree contains a plan which doesn't implement SerializablePlan, there would be a ClassCastException somewhere, e.g. in
@Override
public SerializablePlan getPlanForSerialization() {
return (SerializablePlan) delegate;
}

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't serialize PaginateOperator (to reduce cursor size), so I can't check plan instanceof PaginateOperator here as it was before.

/** No need to serialize a PaginateOperator, it actually does nothing - it is a wrapper. */
@Override
public SerializablePlan getPlanForSerialization() {
return (SerializablePlan) input;
}

That means I have to do another protection here. The incoming tree could be ProjectOperator -> ResourceMonitorPlan -> PrometheusIndexScan or ProjectOperator -> ValuesOperator.

Signed-off-by: Yury-Fridlyand <yury.fridlyand@improving.com>
@Yury-Fridlyand
Copy link
Author

Superseded by opensearch-project#1498

@Yury-Fridlyand Yury-Fridlyand deleted the dev-pagination-v2_revD branch June 24, 2023 00:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants