How do I prevent duplicate Lambda events?

Short description

Preventing duplicate events reduces unnecessary API calls, code process time, data inconsistency, throttles, and latency.

To prevent duplicate Lambda events, you must design your function logic to correctly handle duplicated events. The function's code must validate input events and identify any that it previously processed. Your application's functionality determines the optimal way to write the code.

Resolution

The following example function logic and best practices apply to most scenarios. Use the idempotency utility from Powertools to prevent duplicate events. The idempotency utility is available for Python, JavaScript (TypeScript), .NET, and Java.

Prevent duplicate Lambda event processing

Complete the following steps:

1. Extract the value of a unique attribute of the input event, such as a transaction or purchase ID.
2. Use a conditional write to put the unique key into your Amazon DynamoDB table.
3. For the record attributes, take the following actions:
   Set Status to IN_PROGRESS.
   For in_progress_expiry timestamp, choose current timestamp in milliseconds + the number of milliseconds left before the execution times out. For python, use context.get_remaining_time_in_millis.
   Note: With in_progress_expiry, the Lambda function can securely retry the logic.
   If the PutItem from the preceding step returns ConditionalCheckFailedException, then another invocation is already processing the event or has already completed it.
4. To use the DynamoDB Time To Live (TTL) attribute with expiry, set the attribute to a later time for the value.
5. Update the key record status to COMPLETE.
6. Complete the action.

Note: When you add AWS services to your architecture, you might incur additional costs. For more information, see Amazon DynamoDB pricing and AWS Pricing.

Prevent duplicate events best practices

Complete the following actions:

• Plan your duplicate event handling strategy before you develop your application.
• When your Lambda code processes a duplicate event, make sure that the process ends without an error.
  Note: A process that ends in error might cause services that invoke your function to further retry.
• Configure the Lambda function timeout setting so that the full runtime processes correctly.
  Note: If you use a separate service to persist data and control duplicated events, then you might need to make API calls to HTTPS endpoints. API calls to HTTPS endpoints might require more runtime than the default 3 seconds.
• Test and optimize your function and make sure that you simulate a valid scenario and rate of requests.
  Important: Test and optimize your duplicate event handling logic to prevent timeouts, excessive latency, or bottlenecks.
• To store session data, use a service that's scalable and provides high throughput, such as DynamoDB.

Note: When you make API calls to Amazon Elastic Compute Cloud (Amazon EC2), you can use the clientToken parameter. This parameter makes sure that a mutating API request successfully completes its workflow only once, even when you initiate multiple retries with the same clientToken.

Related information

Understanding the Lambda programming model

Making retries safe with idempotent APIs