AWS Serverless Error Handling for Stream-Based Events
Error Handling for Stream-Based Events
Stream-based event sources must keep record order consistent across shards.
Stream-based event sources could be Kinesis Data Streams or DynamoDB Streams.
If Lambda encounters an issue while processing a batch of data, it will stop processing new data.
A batch of data is a large amount of data.
To discover blocked shards, you can use the Iterator-Age metric.
It measures the age of the stream records your function recently processed.
Failure Management
By using four configuration options, you can manage failures better:
- Bisect batch on function error
- Maximum retry attempts
- Maximum record age
- On-failure destination
If a function error occurs, Lambda will split the batch in half and resume each half separately.
Maximum retry attempts and maximum record age limit the number of retries on a failed batch.
An on-failure destination allows you to send failed records for offline processing.
Error Handling For Stream-Based Events Video
W3schools.com collaborates with Amazon Web Services to deliver digital training content to our students.
How Error Handling for Stream-Based Events Works
When a function produces an error, Lambda divides the batch in half (Bisect batch).
Lambda delivers each half to your function individually, keeping record order.
When Lambda divides a batch, it also resets the retry and max-age parameters.
Lambda will keep dividing the batches until it finds the bad record.
Lambda will keep trying to send the bad record according to the maximum retry attempts.
Lambda will send it to the SNS topic defined for the on-failure destination if it continues to fail.
After removing the erroring record, Lambda goes back through each of the smaller batches it made.
The downside of this process is that some records could be processed multiple times.
It is also called idempotency.
Idempotency must be managed by you.
