-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: update date generator to support backfills #51
Conversation
Aah our integration tests are failing because the "mock Scihub" components don't support S2C...
Fixing... |
@@ -33,7 +33,8 @@ def test_that_link_fetching_invocation_executes_correctly( | |||
queue_url: str, | |||
): | |||
execution_arn = step_function_client.start_execution( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rather than fiddling with the mocked responses I figured we can just request platforms that we know are supported by the mocked API. This will also help us avoid an error when ESA launches Sentinel-2D in >=2028 😄
Probably more importantly this also helps us poke to make sure our StepFunction can handle overriding the "platform" to the "date generator"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great! Nice to see this could be implemented with so few code changes.
Approving, so feel free to ignore my only comment about type annotation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sweet!
What I am changing
This PR updates the functionality of the "date generator" Lambda function to support backfills. Currently the function will always run relative to "now", which is great for forward processing but means we can't invoke this as if we were in the past. Specifically I am making this change to catch ~half of a day of granules from Sentinel-2C on January 21, 2025 that we did not catch in the 5 day lookback we have with our forward processing.
How I did it
I took advantage of the changes @chuckwondo made to facilitate testing the "date generator" without needing to do things like patch
datetime
for time travel. I extended the changes he made to allow these inputs (platforms, "now", and lookback days) to come from the Lambda function payload rather than through kwargs used by unit tests.This allows us to invoke the StepFunction with a payload like,
How you can test it
I updated the unit tests to pass the inputs via event payload rather than extra kwargs,