Add checkpoint mandatory configuration #92
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add new Flint config for checkpoint mandatory option. It is enabled by default which means all incremental refresh (
CREATE
statement withauto_refresh=true
) must providecheckpoint_location
. Doc: https://github.com/dai-chen/opensearch-spark/blob/add-mandatory-checkpoint-option/docs/index.md#configurationsTODO
Currently all Spark streaming job related validation happens when job start. For example, OS index is created even though job start failed due to missing checkpoint location. For this checkpoint validation, it can be solved by building streaming job early. However, other check on table/options maybe performed only when job start. Need to figure out how to validate early in general. Issue: #65
Example
Issues Resolved
#87
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.