You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hey, love the SDK so far, I just had a few questions on designing a SQS message consumer.
I'm assuming the additional functionality I'm about to describe is out of the scope of the SDK but I'm new to Rust and would like a few pointers.
NOTE 1: The consumer is not running on AWS Lambda.
NOTE 2: I'm concerned with long-polling consumers.
1. Heartbeats
The AWS docs suggest using a heartbeat when the processing time for individual messages isn't fixed/known beforehand. In Go I would do something like running a select loop multiplexing between the processing function result and a timer, executing logic to increase the message's visibility timeout in intervals and stopping the select loop once processing was completed. Looks like I could implement something similar in rust (tokio seemed to have similar tools to spawn tasks and select over them) but I was wondering if there was a better pattern I could use.
2. Self-Regulation
What's a good way to slow-down/stop polling (i.e calling ReceiveMessage)? Maybe there's a way where I can introspect the resource utilization on the server but instead I think a simpler alternative is to set a predetermined MAX_CONCURRENCY_LIMIT. The poller abstraction would keep track of the number of messages currently being processed and regulate based on that. Is there a better pattern to prevent self-overloading?
3. IO bound message processing
This is perhaps a pure Async question but for messages whose processing is IO bound, i.e a good fit for the async model, does it matter if I have a single poller requesting large batches with a large request wait time (then delegating message processing to separate tasks) vs having multiple pollers requesting smaller batches and/or with a smaller request wait time?
I understand a lot of the complexity comes from distributed systems and the async programming model, but to me it seems like these problems are general and not domain specific. Would it make sense for the SDK to provide a framework for this? I only ask because it's easy to get the design wrong and I assume many users of SQS would have similar needs.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Hey, love the SDK so far, I just had a few questions on designing a SQS message consumer.
I'm assuming the additional functionality I'm about to describe is out of the scope of the SDK but I'm new to Rust and would like a few pointers.
NOTE 1: The consumer is not running on AWS Lambda.
NOTE 2: I'm concerned with long-polling consumers.
1. Heartbeats
select
loop multiplexing between the processing function result and a timer, executing logic to increase the message's visibility timeout in intervals and stopping theselect
loop once processing was completed. Looks like I could implement something similar in rust (tokio
seemed to have similar tools to spawn tasks and select over them) but I was wondering if there was a better pattern I could use.2. Self-Regulation
ReceiveMessage
)? Maybe there's a way where I can introspect the resource utilization on the server but instead I think a simpler alternative is to set a predeterminedMAX_CONCURRENCY_LIMIT
. Thepoller
abstraction would keep track of the number of messages currently being processed and regulate based on that. Is there a better pattern to prevent self-overloading?3. IO bound message processing
I understand a lot of the complexity comes from distributed systems and the async programming model, but to me it seems like these problems are general and not domain specific. Would it make sense for the SDK to provide a framework for this? I only ask because it's easy to get the design wrong and I assume many users of SQS would have similar needs.
Beta Was this translation helpful? Give feedback.
All reactions