-
Notifications
You must be signed in to change notification settings - Fork 452
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support to retry function execution on invocation failures #6603
Conversation
bb79bd4
to
b698abc
Compare
@mathewc / @brettsam - PR ready for review! @jeffhollan - let me know if there is a doc I can link here. |
src/WebJobs.Script.Abstractions/Description/FunctionMetadata.cs
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Addressed CR feedback!
a9db755
to
2dfc48d
Compare
Dear @pragnagopa and @mathewc, we are currently looking into running full integration tests with pytest through a host instance provided by azure-functions-core-tools. We are probably aiming at similar things which have been outlined within [1] and [2] and already sketched out within [3] and [4] by @jeffhollan and @Francisco-Gamino, but this time for a pure Python environment. As we don't only want to run tests looking for successful outcomes but also want to wrap the test harness around erroneous function invocations, we have been looking for a way to get around the default setting with a retry count of 3, which terminates the host instance after trying to invoke the function three times like
Actually, we would like to reduce that to "1" when running within the test harness in order to speed things up. So, we have been happy to find things like host.json or function.json just recently coming from your pen.
Now - as these new attributes are not mentioned within host.json reference for Azure Functions 2.x and later yet - am I right to assume the corresponding improvements are not part of the most recent release of Azure Functions Core Tools yet? I am currently running
Regarding this, I have two questions:
Please correct me if I am wrong on any of the details I was trying to outline here. With kind regards, [1] https://stackoverflow.com/questions/62749060/unit-and-integration-test-for-azure-function-with-servicebustrigger |
@pragnagopa to help - but I'm not sure if the retry policy here is going to apply to the retry you are showing there (which is retrying communication to a language worker, and not just retrying execution of a message). That said, to answer other questions, yes it's not yet in core tools, docs will merge sometime next week once feature is live and core tools get released. Likely some way to test out with latest WebJobs SDK and such but I'm not sure how. The release is rolling out right now I believe. |
@amotl - @jeffhollan is right. Retry support added in this PR is for function executions. Following log message
is coming from built-in retries to handle process crashes - in this case if python process crashes - functions host retries upto |
Dear @jeffhollan and @pragnagopa, thanks a bunch for your quick answers. What I was trying to achieve is to make the complete function host instance terminate itself by doing such a thing:
In this manner, our idea was to spin up a function/host instance from a pytest fixture, submit a single or more messages to it by e.g. invoking
and then check the outcome of that what has been done when processing the message(s) (e.g. writing to a database). After seeing that the host was shutting down itself after three attempts, we figured there would be a way to get around terminating the process from outside (the pytest fixture) after the test run has completed. Now, after learning what the added "retry" functionality is all about, I believe this was an ill-fated idea. With kind regards, P.S.: One last question, @jeffhollan: I see that you are terminating the function host after 30 seconds like In contrast to this, I am aiming at getting this together more tightly and terminate the function host directly after it has done its job processing all the messages coming from a single function invocation. If you know any other people or resources having attempted similar things running full integration tests through Core Tools' Function Host which could guide me along, please let me know. We are aiming to wrap a test harness around a scenario where messages are ingested from EventHub (probably similar to your example at [1]) and submitted to both CrateDB and Kafka, see also Azure/azure-functions-kafka-extension#181. |
@amotl - looking at the code you shared:
Please open new issue on : https://github.com/Azure/azure-functions-python-worker/issues if you need more guidance how to proceed. |
Dear Pragna, thanks for your answer and sorry for spamming this PR. I believe we will just use Regarding another question to proceed where we are aiming at, I have just created Azure/azure-functions-python-worker#764. Thanks again and with kind regards, |
Issue describing the changes in this PR
Functions host changes to add retry support : Azure/azure-webjobs-sdk#2463
Retry can be added in
host.json
for function app level or can be added infunction.json
for function level. If both are specified,function.json
takes precedence.resolves #6664
Pull request checklist
release_notes.md
Note: Documentation for this feature is tracked internally.