Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: update date generator to support backfills #51

Merged
merged 5 commits into from
Jan 30, 2025

Conversation

ceholden
Copy link
Collaborator

What I am changing

This PR updates the functionality of the "date generator" Lambda function to support backfills. Currently the function will always run relative to "now", which is great for forward processing but means we can't invoke this as if we were in the past. Specifically I am making this change to catch ~half of a day of granules from Sentinel-2C on January 21, 2025 that we did not catch in the 5 day lookback we have with our forward processing.

How I did it

I took advantage of the changes @chuckwondo made to facilitate testing the "date generator" without needing to do things like patch datetime for time travel. I extended the changes he made to allow these inputs (platforms, "now", and lookback days) to come from the Lambda function payload rather than through kwargs used by unit tests.

This allows us to invoke the StepFunction with a payload like,

{
    "now": "2025-01-22",
    "lookback_days": 2,
    "platforms": ["S2B", "S2C"],
    // rest of StepFunction default input payload
    ...,
}

How you can test it

I updated the unit tests to pass the inputs via event payload rather than extra kwargs,

cd lambdas/date_generator
make test

@ceholden
Copy link
Collaborator Author

ceholden commented Jan 29, 2025

Aah our integration tests are failing because the "mock Scihub" components don't support S2C...

[ERROR] FileNotFoundError: [Errno 2] No such file or directory: '/var/task/scihub_responses/scihub_response_platform_S2C_index_1_yesterday.json'
Traceback (most recent call last):
  File "/var/task/handler.py", line 24, in handler
    body = (responses_dir / response_fixture).read_text()
  File "/var/lang/lib/python3.11/pathlib.py", line 1058, in read_text
    with self.open(mode='r', encoding=encoding, errors=errors) as f:
  File "/var/lang/lib/python3.11/pathlib.py", line 1044, in open
    return io.open(self, mode, buffering, encoding, errors, newline)

Fixing...

@@ -33,7 +33,8 @@ def test_that_link_fetching_invocation_executes_correctly(
queue_url: str,
):
execution_arn = step_function_client.start_execution(
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than fiddling with the mocked responses I figured we can just request platforms that we know are supported by the mocked API. This will also help us avoid an error when ESA launches Sentinel-2D in >=2028 😄

Probably more importantly this also helps us poke to make sure our StepFunction can handle overriding the "platform" to the "date generator"

@ceholden ceholden marked this pull request as ready for review January 29, 2025 16:53
chuckwondo
chuckwondo previously approved these changes Jan 29, 2025
Copy link
Collaborator

@chuckwondo chuckwondo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great! Nice to see this could be implemented with so few code changes.

Approving, so feel free to ignore my only comment about type annotation.

lambdas/date_generator/handler.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@chuckwondo chuckwondo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sweet!

@ceholden ceholden merged commit 129b902 into main Jan 30, 2025
3 checks passed
@ceholden ceholden deleted the ceh/allow-param-dategen-via-stepfunc-input branch January 30, 2025 14:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants