AWS Serverless Scaling Considerations for SQS
Scaling Considerations for AWS SQS
The SQS queue buffers API Gateway queries, allowing for asynchronous processing.
As a Lambda event source, AWS SQS controls polling the queue for you.
But you still have control over other performance-related settings.
Scaling Considerations for AWS SQS Video
W3schools.com collaborates with Amazon Web Services to deliver digital training content to our students.
How it Works
Lambda automatically uses five parallel processes to clear the queue.
These five simultaneous processes indicate Lambda is calling your Lambda function five times.
To prevent having your Lambda function throttled, make sure the reserved concurrency is at least five.
If the Lambda service detects an increase in queue size, it raises batch size.
That implies it will call more Lambda functions concurrently.
Lambda will keep adding processes per minute until the queue slows or hits capacity.
Unless otherwise specified, the maximum concurrency is 1000.
Lambda Failures
The Lambda service will reduce the number of processes polling the queue on Lambda function failures.
Failures are placed in the dead-letter queue and deleted from your SQS queue.
If you choose a batch size that is too big, your Lambda function may timeout.
If it happens frequently, Lambda slows polling, causing a queue blockage.
Unprocessed messages
Unprocessed messages become visible if the visibility timeout expires before the Lambda function processes them.
Those messages will need to be processed again, increasing queue size.
This raises the possibility of duplicate message processing.
The ideal approach is to set your visibility timeout to 6 times the function timeout.
It is critical to set the "MaxReceive" count to balance for high concurrency.
It helps with keeping things moving in the queue when sending too many messages to the dead-letter queue.
The table below shows the characteristics of an SQS queue as a Lambda event source.
| Parameter | Value/Limit | How the parameter is set |
|---|---|---|
| Batch message size limit | 1-10 | Configured with event source on the Lambda function |
| Number of default pollers | 5 | Managed by the Lambda service |
| Lambda's rate of increasing parallel pollers | Up to 60 per minute | Managed by the Lambda service |
| Number of concurrent batches Lambda handles | Up to 10000 | Managed by the Lambda service |
| Number of concurrent Lambda functions | The lesser of 1,000, function or account limit | Reserved concurrency limit |
| Messages per queue | No limit | N/A |
| Visibility timeout | 0 seconds to 12 hours | Configured on the queue |
| Number of retries | 1-1000 | Maxreceive count |
| Function timeout | 0 seconds to 15 minutes | Configured on the function |
Related reads:
Consuming Messages Using Long PollingAmazon SQS Limits
Lambda Developer Guide: Using Lambda with Amazon SQS
How to Use Amazon SQS as an Event Source for Lambda
