Add configuration to disable reprocess files cleanup by v1Cleaner

[nirutgupta]

Overview

Issue: Missing records in Snowflake when streaming data using the Snowflake Sink Connector. Customers who have set up multiple topics to map to a single table might see data loss.

Cause: Cleaning up reprocessFiles may remove files that haven’t been fully loaded yet, leading to data loss.

Solution: To avoid this, enable the snowflake.snowpipe.v1Cleaner.disable.reprocessFiles.cleanup flag. This will stop the cleanup of reprocessFiles and help prevent any data loss during the process.

Pre-review checklist

• This change should be part of a Behavior Change Release. See go/behavior-change.
• This change has passed Merge gate tests
• Snowpipe Changes
• Snowpipe Streaming Changes
• This change is TEST-ONLY
• This change is README/Javadocs only
• This change is protected by a config parameter <PARAMETER_NAME> eg snowflake.ingestion.method.
  • Yes - Added end to end and Unit Tests.
  • No - Suggest why it is not param protected
• Is his change protected by parameter <PARAMETER_NAME> on the server side?
  • The parameter/feature is not yet active in production (partial rollout or PrPr, see Changes for Unreleased Features and Fixes).
  • If there is an issue, it can be safely mitigated by turning the parameter off. This is also verified by a test (See go/ppp).