Idempotency

The idempotency utility provides a simple solution to convert your Lambda functions into idempotent operations which are safe to retry.

Key features¶

Prevent Lambda handler from executing more than once on the same event payload during a time window
Ensure Lambda handler returns the same result when called with the same payload
Select a subset of the event as the idempotency key using JMESPath expressions
Set a time window in which records with the same payload should be considered duplicates
Expires in-progress executions if the Lambda function times out halfway through

Terminology¶

The property of idempotency means that an operation does not cause additional side effects if it is called more than once with the same input parameters.

Idempotent operations will return the same result when they are called multiple times with the same parameters. This makes idempotent operations safe to retry.

Idempotency key is a hash representation of either the entire event or a specific configured subset of the event, and invocation results are JSON serialized and stored in your persistence storage layer.

Idempotency record is the data representation of an idempotent request saved in your preferred storage layer. We use it to coordinate whether a request is idempotent, whether it's still valid or expired based on timestamps, etc.

classDiagram
    direction LR
    class IdempotencyRecord {
        idempotencyKey string
        status Status
        expiryTimestamp number
        inProgressExpiryTimestamp number
        responseData Json~string~
        payloadHash string
    }
    class Status {
        <<Enumeration>>
        INPROGRESS
        COMPLETE
        EXPIRED internal_only
    }
    IdempotencyRecord -- Status

Idempotency record representation

Getting started¶

Installation¶

Install the library in your project

npm i @aws-lambda-powertools/idempotency @aws-sdk/client-dynamodb @aws-sdk/lib-dynamodb

While we support Amazon DynamoDB as a persistence layer out of the box, you need to bring your own AWS SDK for JavaScript v3 DynamoDB client.

Note

This utility supports AWS SDK for JavaScript v3 only. If you are using the nodejs18.x runtime or newer, the AWS SDK for JavaScript v3 is already installed and you can install only the utility.

IAM Permissions¶

Your Lambda function IAM Role must have dynamodb:GetItem, dynamodb:PutItem, dynamodb:UpdateItem and dynamodb:DeleteItem IAM permissions before using this feature. If you're using one of our examples: AWS Serverless Application Model (SAM) or Terraform the required permissions are already included.

Required resources¶

Before getting started, you need to create a persistent storage layer where the idempotency utility can store its state - your lambda functions will need read and write access to it.

As of now, Amazon DynamoDB is the only supported persistent storage layer, so you'll need to create a table first.

Default table configuration

If you're not changing the default configuration for the DynamoDB persistence layer, this is the expected default configuration:

Configuration	Default value	Notes
Partition key	`id`	The id of each idempotency record which a combination of `functionName#hashOfPayload`.
TTL attribute name	`expiration`	This can only be configured after your table is created if you're using AWS Console.

Tip: You can share a single state table for all functions

You can reuse the same DynamoDB table to store idempotency state. We add the Lambda function name in addition to the idempotency key as a hash key.

AWS Cloud Development Kit (CDK) exampleAWS Serverless Application Model (SAM) exampleTerraform example

template.ts
import { Stack, type StackProps } from 'aws-cdk-lib';
import { AttributeType, BillingMode, Table } from 'aws-cdk-lib/aws-dynamodb';
import { Runtime } from 'aws-cdk-lib/aws-lambda';
import { NodejsFunction } from 'aws-cdk-lib/aws-lambda-nodejs';
import type { Construct } from 'constructs';

export class IdempotencyStack extends Stack {
  public constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    const table = new Table(this, 'idempotencyTable', {
      partitionKey: {
        name: 'id',
        type: AttributeType.STRING,
      },
      timeToLiveAttribute: 'expiration',
      billingMode: BillingMode.PAY_PER_REQUEST,
    });

    const fnHandler = new NodejsFunction(this, 'helloWorldFunction', {
      runtime: Runtime.NODEJS_20_X,
      handler: 'handler',
      entry: 'src/index.ts',
      environment: {
        IDEMPOTENCY_TABLE_NAME: table.tableName,
      },
    });
    table.grantReadWriteData(fnHandler);
  }
}

template.yaml
Transform: AWS::Serverless-2016-10-31
Resources:
  IdempotencyTable:
    Type: AWS::DynamoDB::Table
    Properties:
      AttributeDefinitions:
        - AttributeName: id
          AttributeType: S
      KeySchema:
        - AttributeName: id
          KeyType: HASH
      TimeToLiveSpecification:
        AttributeName: expiration
        Enabled: true
      BillingMode: PAY_PER_REQUEST

  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: nodejs20.x
      Handler: app.py
      Policies:
        - Statement:
            - Sid: AllowDynamodbReadWrite
              Effect: Allow
              Action:
                - dynamodb:PutItem
                - dynamodb:GetItem
                - dynamodb:UpdateItem
                - dynamodb:DeleteItem
              Resource: !GetAtt IdempotencyTable.Arn

template.tf
terraform {
    required_providers {
        aws = {
        source  = "hashicorp/aws"
        version = "~> 4.0"
        }
    }
}

provider "aws" {
    region = "us-east-1" # Replace with your desired AWS region
}

resource "aws_dynamodb_table" "IdempotencyTable" {
    name         = "IdempotencyTable"
    billing_mode = "PAY_PER_REQUEST"
    hash_key     = "id"
    attribute {
        name = "id"
        type = "S"
    }
    ttl {
        attribute_name = "expiration"
        enabled        = true
    }
}

resource "aws_lambda_function" "IdempotencyFunction" {
    function_name = "IdempotencyFunction"
    role          = aws_iam_role.IdempotencyFunctionRole.arn
    runtime       = "nodejs20.x"
    handler       = "index.handler"
    filename      = "lambda.zip"
}

resource "aws_iam_role" "IdempotencyFunctionRole" {
    name = "IdempotencyFunctionRole"

    assume_role_policy = jsonencode({
        Version = "2012-10-17"
        Statement = [
        {
            Sid    = ""
            Effect = "Allow"
            Principal = {
            Service = "lambda.amazonaws.com"
            }
            Action = "sts:AssumeRole"
        },
        ]
    })
}

resource "aws_iam_policy" "LambdaDynamoDBPolicy" {
    name        = "LambdaDynamoDBPolicy"
    description = "IAM policy for Lambda function to access DynamoDB"
    policy = jsonencode({
        Version = "2012-10-17"
        Statement = [
        {
            Sid    = "AllowDynamodbReadWrite"
            Effect = "Allow"
            Action = [
            "dynamodb:PutItem",
            "dynamodb:GetItem",
            "dynamodb:UpdateItem",
            "dynamodb:DeleteItem",
            ]
            Resource = aws_dynamodb_table.IdempotencyTable.arn
        },
        ]
    })
}

resource "aws_iam_role_policy_attachment" "IdempotencyFunctionRoleAttachment" {
    role       = aws_iam_role.IdempotencyFunctionRole.name
    policy_arn = aws_iam_policy.LambdaDynamoDBPolicy.arn
}

Warning: Large responses with DynamoDB persistence layer

When using this utility with DynamoDB, your function's responses must be smaller than 400KB.

Larger items cannot be written to DynamoDB and will cause exceptions.

Info: DynamoDB

Each function invocation will make only 1 request to DynamoDB by using DynamoDB's conditional expressions to ensure that we don't overwrite existing records, and ReturnValuesOnConditionCheckFailure to return the record if it exists. See AWS Blog post on handling conditional write errors for more details. For retried invocations, you will see 1WCU and 1RCU. Review the DynamoDB pricing documentation to estimate the cost.

MakeIdempotent function wrapper¶

You can quickly start by initializing the DynamoDBPersistenceLayer class and using it with the makeIdempotent function wrapper on your Lambda handler.

index.tstypes.ts

import { randomUUID } from 'node:crypto';
import { makeIdempotent } from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import type { Context } from 'aws-lambda';
import type { Request, Response, SubscriptionResult } from './types.js';

const persistenceStore = new DynamoDBPersistenceLayer({
  tableName: 'idempotencyTableName',
});

const createSubscriptionPayment = async (
  event: Request
): Promise<SubscriptionResult> => {
  // ... create payment
  return {
    id: randomUUID(),
    productId: event.productId,
  };
};

export const handler = makeIdempotent(
  async (event: Request, _context: Context): Promise<Response> => {
    try {
      const payment = await createSubscriptionPayment(event);

      return {
        paymentId: payment.id,
        message: 'success',
        statusCode: 200,
      };
    } catch (error) {
      throw new Error('Error creating payment');
    }
  },
  {
    persistenceStore,
  }
);

export type Request = {
  user: string;
  productId: string;
};

export type Response = {
  [key: string]: unknown;
};

export type SubscriptionResult = {
  id: string;
  productId: string;
};

After processing this request successfully, a second request containing the exact same payload above will now return the same response, ensuring our customer isn't charged twice.

Note

In this example, the entire Lambda handler is treated as a single idempotent operation. If your Lambda handler can cause multiple side effects, or you're only interested in making a specific logic idempotent, use the makeIdempotent high-order function only on the function that needs to be idempotent.

See Choosing a payload subset for idempotency for more elaborate use cases.

You can also use the makeIdempotent function wrapper on any method that returns a response to make it idempotent. This is useful when you want to make a specific logic idempotent, for example when your Lambda handler performs multiple side effects and you only want to make a specific one idempotent.

Limitation

Make sure to return a JSON serializable response from your function, otherwise you'll get an error.

When using makeIdempotent on arbitrary functions, you can tell us which argument in your function signature has the data we should use via dataIndexArgument. If you don't specify this argument, we'll use the first argument in the function signature.

index.tstypes.ts

import { randomUUID } from 'node:crypto';
import {
  IdempotencyConfig,
  makeIdempotent,
} from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import type { Context } from 'aws-lambda';
import type { Request, Response, SubscriptionResult } from './types.js';

const persistenceStore = new DynamoDBPersistenceLayer({
  tableName: 'idempotencyTableName',
});
const config = new IdempotencyConfig({});

const reportSubscriptionMetrics = async (
  _transactionId: string,
  _user: string
): Promise<void> => {
  // ... send notification
};

const createSubscriptionPayment = makeIdempotent(
  async (
    transactionId: string,
    event: Request
  ): Promise<SubscriptionResult> => {
    // ... create payment
    return {
      id: transactionId,
      productId: event.productId,
    };
  },
  {
    persistenceStore,
    dataIndexArgument: 1,
    config,
  }
);

export const handler = async (
  event: Request,
  context: Context
): Promise<Response> => {
  config.registerLambdaContext(context);
  try {
    const transactionId = randomUUID();
    const payment = await createSubscriptionPayment(transactionId, event);

    await reportSubscriptionMetrics(transactionId, event.user);

    return {
      paymentId: payment.id,
      message: 'success',
      statusCode: 200,
    };
  } catch (error) {
    throw new Error('Error creating payment');
  }
};

export type Request = {
  user: string;
  productId: string;
};

export type Response = {
  [key: string]: unknown;
};

export type SubscriptionResult = {
  id: string;
  productId: string;
};

The function this example has two arguments, note that while wrapping it with the makeIdempotent high-order function, we specify the dataIndexArgument as 1 to tell the decorator that the second argument is the one with the data we should use to make the function idempotent. Remember that arguments are zero-indexed, so the first argument is 0, the second is 1, etc.

Idempotent Decorator¶

You can also use the @idempotent decorator to make your Lambda handler idempotent, similar to the makeIdempotent function wrapper.

Info

The class method decorators in this project follow the experimental implementation enabled via the experimentalDecorators compiler option in TypeScript.

Additionally, they are implemented to decorate async methods. When decorating a synchronous one, the decorator replaces its implementation with an async one causing the caller to have to await the now decorated method.

If this is not the desired behavior, you can use one of the other patterns to make your logic idempotent.

index.tstypes.ts

import type { LambdaInterface } from '@aws-lambda-powertools/commons/types';
import {
  IdempotencyConfig,
  idempotent,
} from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import type { Context } from 'aws-lambda';
import type { Request, Response } from './types.js';

const dynamoDBPersistenceLayer = new DynamoDBPersistenceLayer({
  tableName: 'idempotencyTableName',
});

const config = new IdempotencyConfig({});

class MyLambda implements LambdaInterface {
  @idempotent({ persistenceStore: dynamoDBPersistenceLayer, config: config })
  public async handler(_event: Request, _context: Context): Promise<Response> {
    // ... process your event
    return {
      message: 'success',
      statusCode: 200,
    };
  }
}

const defaultLambda = new MyLambda();
export const handler = defaultLambda.handler.bind(defaultLambda);

import type { IdempotencyRecordStatusValue } from '@aws-lambda-powertools/idempotency/types';

export type Request = {
  user: string;
  productId: string;
};

export type Response = {
  [key: string]: unknown;
};

export type SubscriptionResult = {
  id: string;
  productId: string;
};

export type ApiSecret = {
  apiKey: string;
  refreshToken: string;
  validUntil: number;
  restEndpoint: string;
};

export type ProviderItem = {
  validation?: string;
  in_progress_expiration?: number;
  status: IdempotencyRecordStatusValue;
  data: string;
};

You can use the decorator on your Lambda handler or on any function that returns a response to make it idempotent. This is useful when you want to make a specific logic idempotent, for example when your Lambda handler performs multiple side effects and you only want to make a specific one idempotent. The configuration options for the @idempotent decorator are the same as the ones for the makeIdempotent function wrapper.

MakeHandlerIdempotent Middy middleware¶

A note about Middy

We guarantee support for both Middy.js v4.x & v5.x with the latter being available only if you are using ES modules. Check their docs to learn more about Middy and its middleware stack as well as best practices when working with Powertools.

If you are using Middy.js as your middleware engine, you can use the makeHandlerIdempotent middleware to make your Lambda handler idempotent.

Similar to the makeIdempotent function wrapper, you can quickly make your Lambda handler idempotent by initializing the DynamoDBPersistenceLayer class and using it with the makeHandlerIdempotent middleware.

index.tstypes.ts

import { randomUUID } from 'node:crypto';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import { makeHandlerIdempotent } from '@aws-lambda-powertools/idempotency/middleware';
import middy from '@middy/core';
import type { Context } from 'aws-lambda';
import type { Request, Response, SubscriptionResult } from './types.js';

const persistenceStore = new DynamoDBPersistenceLayer({
  tableName: 'idempotencyTableName',
});

const createSubscriptionPayment = async (
  event: Request
): Promise<SubscriptionResult> => {
  // ... create payment
  return {
    id: randomUUID(),
    productId: event.productId,
  };
};

export const handler = middy(
  async (event: Request, _context: Context): Promise<Response> => {
    try {
      const payment = await createSubscriptionPayment(event);

      return {
        paymentId: payment.id,
        message: 'success',
        statusCode: 200,
      };
    } catch (error) {
      throw new Error('Error creating payment');
    }
  }
).use(
  makeHandlerIdempotent({
    persistenceStore,
  })
);

export type Request = {
  user: string;
  productId: string;
};

export type Response = {
  [key: string]: unknown;
};

export type SubscriptionResult = {
  id: string;
  productId: string;
};

For the middleware to work, your Lambda function handler must return a value different from undefined. This is a known limitation of the early return feature in Middy.js. If your use case requires early returns, you can use the makeIdempotent function wrapper instead.

Choosing a payload subset for idempotency¶

Use IdempotencyConfig to instruct the idempotent decorator to only use a portion of your payload to verify whether a request is idempotent, and therefore it should not be retried. When dealing with a more elaborate payload, where parts of the payload always change, you should use the eventKeyJmesPath parameter.

Payment scenario

In this example, we have a Lambda handler that creates a payment for a user subscribing to a product. We want to ensure that we don't accidentally charge our customer by subscribing them more than once.

Imagine the function executes successfully, but the client never receives the response due to a connection issue. It is safe to retry in this instance, as the idempotent decorator will return a previously saved response.

What we want here is to instruct Idempotency to use the user and productId fields from our incoming payload as our idempotency key. If we were to treat the entire request as our idempotency key, a simple HTTP header or timestamp change would cause our customer to be charged twice.

Deserializing JSON strings in payloads for increased accuracy.

The payload extracted by the eventKeyJmesPath is treated as a string by default. This means there could be differences in whitespace even when the JSON payload itself is identical.

To alter this behaviour, we can use the JMESPath built-in function powertools_json() to treat the payload as a JSON object rather than a string.

index.tsExample eventtypes.ts

import { randomUUID } from 'node:crypto';
import {
  IdempotencyConfig,
  makeIdempotent,
} from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import type { Context } from 'aws-lambda';
import type { Request, Response, SubscriptionResult } from './types.js';

const persistenceStore = new DynamoDBPersistenceLayer({
  tableName: 'idempotencyTableName',
});

const createSubscriptionPayment = async (
  _user: string,
  productId: string
): Promise<SubscriptionResult> => {
  // ... create payment
  return {
    id: randomUUID(),
    productId: productId,
  };
};

// Deserialize JSON string under the "body" key, then extract the "user" and "productId" keys
const config = new IdempotencyConfig({
  eventKeyJmesPath: 'powertools_json(body).["user", "productId"]',
});

export const handler = makeIdempotent(
  async (event: Request, _context: Context): Promise<Response> => {
    try {
      const payment = await createSubscriptionPayment(
        event.user,
        event.productId
      );

      return {
        paymentId: payment.id,
        message: 'success',
        statusCode: 200,
      };
    } catch (error) {
      throw new Error('Error creating payment');
    }
  },
  {
    persistenceStore,
    config,
  }
);

{
  "version": "2.0",
  "routeKey": "ANY /createpayment",
  "rawPath": "/createpayment",
  "rawQueryString": "",
  "headers": {
    "Header1": "value1",
    "X-Idempotency-Key": "abcdefg"
  },
  "requestContext": {
    "accountId": "123456789012",
    "apiId": "api-id",
    "domainName": "id.execute-api.us-east-1.amazonaws.com",
    "domainPrefix": "id",
    "http": {
      "method": "POST",
      "path": "/createpayment",
      "protocol": "HTTP/1.1",
      "sourceIp": "ip",
      "userAgent": "agent"
    },
    "requestId": "id",
    "routeKey": "ANY /createpayment",
    "stage": "$default",
    "time": "10/Feb/2021:13:40:43 +0000",
    "timeEpoch": 1612964443723
  },
  "body": "{\"user\":\"xyz\",\"productId\":\"123456789\"}",
  "isBase64Encoded": false
}

export type Request = {
  user: string;
  productId: string;
};

export type Response = {
  [key: string]: unknown;
};

export type SubscriptionResult = {
  id: string;
  productId: string;
};

Lambda timeouts¶

To prevent against extended failed retries when a Lambda function times out, Powertools for AWS Lambda calculates and includes the remaining invocation available time as part of the idempotency record. This is automatically done when you wrap your Lambda handler with the makeIdempotent function wrapper, or use the makeHandlerIdempotent Middy middleware.

Example

If a second invocation happens after this timestamp, and the record is marked as INPROGRESS, we will execute the invocation again as if it was in the EXPIRED state (e.g, expire_seconds field elapsed).

This means that if an invocation expired during execution, it will be quickly executed again on the next retry.

Important

If you are only using the makeIdempotent function wrapper to guard isolated parts of your code outside of your handler, you must use registerLambdaContext available in the idempotency config object to benefit from this protection.

Here is an example on how you register the Lambda context in your handler:

Registering Lambda Context

import { randomUUID } from 'node:crypto';
import {
  IdempotencyConfig,
  makeIdempotent,
} from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import type { Context } from 'aws-lambda';
import type { Request, Response, SubscriptionResult } from './types.js';

const persistenceStore = new DynamoDBPersistenceLayer({
  tableName: 'idempotencyTableName',
});
const config = new IdempotencyConfig({});

const createSubscriptionPayment = makeIdempotent(
  async (
    transactionId: string,
    event: Request
  ): Promise<SubscriptionResult> => {
    // ... create payment
    return {
      id: transactionId,
      productId: event.productId,
    };
  },
  {
    persistenceStore,
    dataIndexArgument: 1,
    config,
  }
);

export const handler = async (
  event: Request,
  context: Context
): Promise<Response> => {
  // Register the Lambda context to the IdempotencyConfig instance
  config.registerLambdaContext(context);
  try {
    const transactionId = randomUUID();
    const payment = await createSubscriptionPayment(transactionId, event);

    return {
      paymentId: payment.id,
      message: 'success',
      statusCode: 200,
    };
  } catch (error) {
    throw new Error('Error creating payment');
  }
};

Handling exceptions¶

If you are making on your entire Lambda handler idempotent, any unhandled exceptions that are raised during the code execution will cause the record in the persistence layer to be deleted. This means that new invocations will execute your code again despite having the same payload. If you don't want the record to be deleted, you need to catch exceptions within the idempotent function and return a successful response.

sequenceDiagram
    autonumber
    participant Client
    participant Lambda
    participant Persistence Layer
    Client->>Lambda: Invoke (event)
    Lambda->>Persistence Layer: Get or set (id=event.search(payload))
    activate Persistence Layer
    Note right of Persistence Layer: Locked during this time. Prevents multiple<br/>Lambda invocations with the same<br/>payload running concurrently.
    Lambda--xLambda: Call handler (event).<br/>Raises exception
    Lambda->>Persistence Layer: Delete record (id=event.search(payload))
    deactivate Persistence Layer
    Lambda-->>Client: Return error response

Idempotent sequence exception

If you are using makeIdempotent on any other function, any unhandled exceptions that are thrown inside the wrapped function will cause the record in the persistence layer to be deleted, and allow the function to be executed again if retried.

If an error is thrown outside the scope of the decorated function and after your function has been called, the persistent record will not be affected. In this case, idempotency will be maintained for your decorated function. Example:

Handling exceptions

import { randomUUID } from 'node:crypto';
import {
  IdempotencyConfig,
  makeIdempotent,
} from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import type { Context } from 'aws-lambda';
import type { Request, Response, SubscriptionResult } from './types.js';

const persistenceStore = new DynamoDBPersistenceLayer({
  tableName: 'idempotencyTableName',
});
const config = new IdempotencyConfig({});

const createSubscriptionPayment = makeIdempotent(
  async (
    transactionId: string,
    event: Request
  ): Promise<SubscriptionResult> => {
    // ... create payment
    return {
      id: transactionId,
      productId: event.productId,
    };
  },
  {
    persistenceStore,
    dataIndexArgument: 1,
    config,
  }
);

export const handler = async (
  event: Request,
  context: Context
): Promise<Response> => {
  config.registerLambdaContext(context);
  /**
   * If an exception is thrown before the wrapped function is called,
   * no idempotency record is created.
   */
  try {
    const transactionId = randomUUID();
    const payment = await createSubscriptionPayment(transactionId, event);

    /**
     * If an exception is thrown after the wrapped function is called,
     * the idempotency record won't be affected so it's safe to retry.
     */

    return {
      paymentId: payment.id,
      message: 'success',
      statusCode: 200,
    };
  } catch (error) {
    throw new Error('Error creating payment');
  }
};

Warning

We will throw IdempotencyPersistenceLayerError if any of the calls to the persistence layer fail unexpectedly.

As this happens outside the scope of your decorated function, you are not able to catch it when making your Lambda handler idempotent.

Idempotency request flow¶

The following sequence diagrams explain how the Idempotency feature behaves under different scenarios.

Successful request¶

sequenceDiagram
    autonumber
    participant Client
    participant Lambda
    participant Persistence Layer
    alt initial request
        Client->>Lambda: Invoke (event)
        Lambda->>Persistence Layer: Get or set idempotency_key=hash(payload)
        activate Persistence Layer
        Note over Lambda,Persistence Layer: Set record status to INPROGRESS. <br> Prevents concurrent invocations <br> with the same payload
        Lambda-->>Lambda: Call your function
        Lambda->>Persistence Layer: Update record with result
        deactivate Persistence Layer
        Persistence Layer-->>Persistence Layer: Update record
        Note over Lambda,Persistence Layer: Set record status to COMPLETE. <br> New invocations with the same payload <br> now return the same result
        Lambda-->>Client: Response sent to client
    else retried request
        Client->>Lambda: Invoke (event)
        Lambda->>Persistence Layer: Get or set idempotency_key=hash(payload)
        activate Persistence Layer
        Persistence Layer-->>Lambda: Already exists in persistence layer.
        deactivate Persistence Layer
        Note over Lambda,Persistence Layer: Record status is COMPLETE and not expired
        Lambda-->>Client: Same response sent to client
    end

Idempotent successful request

Successful request with cache enabled¶

In-memory cache is disabled by default.

sequenceDiagram
    autonumber
    participant Client
    participant Lambda
    participant Persistence Layer
    alt initial request
      Client->>Lambda: Invoke (event)
      Lambda->>Persistence Layer: Get or set idempotency_key=hash(payload)
      activate Persistence Layer
      Note over Lambda,Persistence Layer: Set record status to INPROGRESS. <br> Prevents concurrent invocations <br> with the same payload
      Lambda-->>Lambda: Call your function
      Lambda->>Persistence Layer: Update record with result
      deactivate Persistence Layer
      Persistence Layer-->>Persistence Layer: Update record
      Note over Lambda,Persistence Layer: Set record status to COMPLETE. <br> New invocations with the same payload <br> now return the same result
      Lambda-->>Lambda: Save record and result in memory
      Lambda-->>Client: Response sent to client
    else retried request
      Client->>Lambda: Invoke (event)
      Lambda-->>Lambda: Get idempotency_key=hash(payload)
      Note over Lambda,Persistence Layer: Record status is COMPLETE and not expired
      Lambda-->>Client: Same response sent to client
    end

Idempotent successful request cached

Successful request with responseHook configured¶

sequenceDiagram
    participant Client
    participant Lambda
    participant Response hook
    participant Persistence Layer
    alt initial request
        Client->>Lambda: Invoke (event)
        Lambda->>Persistence Layer: Get or set idempotency_key=hash(payload)
        activate Persistence Layer
        Note over Lambda,Persistence Layer: Set record status to INPROGRESS. <br> Prevents concurrent invocations <br> with the same payload
        Lambda-->>Lambda: Call your function
        Lambda->>Persistence Layer: Update record with result
        deactivate Persistence Layer
        Persistence Layer-->>Persistence Layer: Update record
        Note over Lambda,Persistence Layer: Set record status to COMPLETE. <br> New invocations with the same payload <br> now return the same result
        Lambda-->>Client: Response sent to client
    else retried request
        Client->>Lambda: Invoke (event)
        Lambda->>Persistence Layer: Get or set idempotency_key=hash(payload)
        activate Persistence Layer
        Persistence Layer-->>Response hook: Already exists in persistence layer.
        deactivate Persistence Layer
        Note over Response hook,Persistence Layer: Record status is COMPLETE and not expired
        Response hook->>Lambda: Response hook invoked
        Lambda-->>Client: Manipulated idempotent response sent to client
    end

Successful idempotent request with a response hook

Expired idempotency records¶

sequenceDiagram
    autonumber
    participant Client
    participant Lambda
    participant Persistence Layer
    alt initial request
        Client->>Lambda: Invoke (event)
        Lambda->>Persistence Layer: Get or set idempotency_key=hash(payload)
        activate Persistence Layer
        Note over Lambda,Persistence Layer: Set record status to INPROGRESS. <br> Prevents concurrent invocations <br> with the same payload
        Lambda-->>Lambda: Call your function
        Lambda->>Persistence Layer: Update record with result
        deactivate Persistence Layer
        Persistence Layer-->>Persistence Layer: Update record
        Note over Lambda,Persistence Layer: Set record status to COMPLETE. <br> New invocations with the same payload <br> now return the same result
        Lambda-->>Client: Response sent to client
    else retried request
        Client->>Lambda: Invoke (event)
        Lambda->>Persistence Layer: Get or set idempotency_key=hash(payload)
        activate Persistence Layer
        Persistence Layer-->>Lambda: Already exists in persistence layer.
        deactivate Persistence Layer
        Note over Lambda,Persistence Layer: Record status is COMPLETE but expired hours ago
        loop Repeat initial request process
            Note over Lambda,Persistence Layer: 1. Set record to INPROGRESS, <br> 2. Call your function, <br> 3. Set record to COMPLETE
        end
        Lambda-->>Client: Same response sent to client
    end

Previous Idempotent request expired

Concurrent identical in-flight requests¶

sequenceDiagram
    autonumber
    participant Client
    participant Lambda
    participant Persistence Layer
    Client->>Lambda: Invoke (event)
    Lambda->>Persistence Layer: Get or set idempotency_key=hash(payload)
    activate Persistence Layer
    Note over Lambda,Persistence Layer: Set record status to INPROGRESS. <br> Prevents concurrent invocations <br> with the same payload
      par Second request
          Client->>Lambda: Invoke (event)
          Lambda->>Persistence Layer: Get or set idempotency_key=hash(payload)
          Lambda--xLambda: IdempotencyAlreadyInProgressError
          Lambda->>Client: Error sent to client if unhandled
      end
    Lambda-->>Lambda: Call your function
    Lambda->>Persistence Layer: Update record with result
    deactivate Persistence Layer
    Persistence Layer-->>Persistence Layer: Update record
    Note over Lambda,Persistence Layer: Set record status to COMPLETE. <br> New invocations with the same payload <br> now return the same result
    Lambda-->>Client: Response sent to client

Concurrent identical in-flight requests

Lambda request timeout¶

sequenceDiagram
    autonumber
    participant Client
    participant Lambda
    participant Persistence Layer
    alt initial request
        Client->>Lambda: Invoke (event)
        Lambda->>Persistence Layer: Get or set idempotency_key=hash(payload)
        activate Persistence Layer
        Note over Lambda,Persistence Layer: Set record status to INPROGRESS. <br> Prevents concurrent invocations <br> with the same payload
        Lambda-->>Lambda: Call your function
        Note right of Lambda: Time out
        Lambda--xLambda: Time out error
        Lambda-->>Client: Return error response
        deactivate Persistence Layer
    else retry after Lambda timeout elapses
        Client->>Lambda: Invoke (event)
        Lambda->>Persistence Layer: Get or set idempotency_key=hash(payload)
        activate Persistence Layer
        Note over Lambda,Persistence Layer: Set record status to INPROGRESS. <br> Reset in_progress_expiry attribute
        Lambda-->>Lambda: Call your function
        Lambda->>Persistence Layer: Update record with result
        deactivate Persistence Layer
        Persistence Layer-->>Persistence Layer: Update record
        Lambda-->>Client: Response sent to client
    end

Idempotent request during and after Lambda timeouts

Optional idempotency key¶

sequenceDiagram
    autonumber
    participant Client
    participant Lambda
    participant Persistence Layer
    alt request with idempotency key
        Client->>Lambda: Invoke (event)
        Lambda->>Persistence Layer: Get or set idempotency_key=hash(payload)
        activate Persistence Layer
        Note over Lambda,Persistence Layer: Set record status to INPROGRESS. <br> Prevents concurrent invocations <br> with the same payload
        Lambda-->>Lambda: Call your function
        Lambda->>Persistence Layer: Update record with result
        deactivate Persistence Layer
        Persistence Layer-->>Persistence Layer: Update record
        Note over Lambda,Persistence Layer: Set record status to COMPLETE. <br> New invocations with the same payload <br> now return the same result
        Lambda-->>Client: Response sent to client
    else request(s) without idempotency key
        Client->>Lambda: Invoke (event)
        Note over Lambda: Idempotency key is missing
        Note over Persistence Layer: Skips any operation to fetch, update, and delete
        Lambda-->>Lambda: Call your function
        Lambda-->>Client: Response sent to client
    end

Optional idempotency key

Advanced¶

Persistence layers¶

DynamoDBPersistenceLayer¶

This persistence layer is built-in, and you can either use an existing DynamoDB table or create a new one dedicated for idempotency state (recommended).

Customizing DynamoDBPersistenceLayer to suit your table structure

import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import { makeHandlerIdempotent } from '@aws-lambda-powertools/idempotency/middleware';
import middy from '@middy/core';
import type { Context } from 'aws-lambda';
import type { Request, Response } from './types.js';

const persistenceStore = new DynamoDBPersistenceLayer({
  tableName: 'idempotencyTableName',
  keyAttr: 'idempotencyKey',
  expiryAttr: 'expiresAt',
  inProgressExpiryAttr: 'inProgressExpiresAt',
  statusAttr: 'currentStatus',
  dataAttr: 'resultData',
  validationKeyAttr: 'validationKey',
});

export const handler = middy(
  async (_event: Request, _context: Context): Promise<Response> => {
    try {
      // ... create payment

      return {
        paymentId: '1234567890',
        message: 'success',
        statusCode: 200,
      };
    } catch (error) {
      throw new Error('Error creating payment');
    }
  }
).use(
  makeHandlerIdempotent({
    persistenceStore,
  })
);

When using DynamoDB as a persistence layer, you can alter the attribute names by passing these parameters when initializing the persistence layer:

Parameter	Default	Description
tableName		Table name to store state
keyAttr	`id`	Partition key of the table. Hashed representation of the payload (unless sort_key_attr is specified)
expiryAttr	`expiration`	Unix timestamp of when record expires
inProgressExpiryAttr	`in_progress_expiration`	Unix timestamp of when record expires while in progress (in case of the invocation times out)
statusAttr	`status`	Stores status of the lambda execution during and after invocation
dataAttr	`data`	Stores results of successfully executed Lambda handlers
validationKeyAttr	`validation`	Hashed representation of the parts of the event used for validation
sortKeyAttr		Sort key of the table (if table is configured with a sort key).
staticPkValue	`idempotency#{LAMBDA_FUNCTION_NAME}`	Static value to use as the partition key. Only used when sort_key_attr is set.

Customizing the default behavior¶

Idempotent decorator can be further configured with IdempotencyConfig as seen in the previous examples. These are the available options for further configuration

Parameter	Default	Description
eventKeyJmespath	`''`	JMESPath expression to extract the idempotency key from the event record using built-in functions
payloadValidationJmespath	`''`	JMESPath expression to validate that the specified fields haven't changed across requests for the same idempotency key e.g., payload tampering.
jmesPathOptions	`undefined`	Custom JMESPath functions to use when parsing the JMESPath expressions. See Custom JMESPath Functions
throwOnNoIdempotencyKey	`false`	Throw an error if no idempotency key was found in the request
expiresAfterSeconds	3600	The number of seconds to wait before a record is expired, allowing a new transaction with the same idempotency key
useLocalCache	`false`	Whether to cache idempotency results in-memory to save on persistence storage latency and costs
localCacheMaxItems	256	Max number of items to store in local cache
hashFunction	`md5`	Function to use for calculating hashes, as provided by the crypto module in the standard library.
responseHook	`undefined`	Function to use for processing the stored Idempotent response. This function hook is called when an existing idempotent response is found. See Manipulating The Idempotent Response

Handling concurrent executions with the same payload¶

This utility will throw an IdempotencyAlreadyInProgressError error if you receive multiple invocations with the same payload while the first invocation hasn't completed yet.

Info

If you receive IdempotencyAlreadyInProgressError, you can safely retry the operation.

This is a locking mechanism for correctness. Since we don't know the result from the first invocation yet, we can't safely allow another concurrent execution.

Using in-memory cache¶

By default, in-memory local caching is disabled, since we don't know how much memory you consume per invocation compared to the maximum configured in your Lambda function.

Note: This in-memory cache is local to each Lambda execution environment

This means it will be effective in cases where your function's concurrency is low in comparison to the number of "retry" invocations with the same payload, because cache might be empty.

You can enable in-memory caching with the useLocalCache parameter:

Caching idempotent transactions in-memory to prevent multiple calls to storage

import { IdempotencyConfig } from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import { makeHandlerIdempotent } from '@aws-lambda-powertools/idempotency/middleware';
import middy from '@middy/core';
import type { Context } from 'aws-lambda';
import type { Request, Response } from './types.js';

const persistenceStore = new DynamoDBPersistenceLayer({
  tableName: 'idempotencyTableName',
});
const config = new IdempotencyConfig({
  useLocalCache: true,
  maxLocalCacheSize: 512,
});

export const handler = middy(
  async (_event: Request, _context: Context): Promise<Response> => {
    try {
      // ... create payment

      return {
        paymentId: '1234567890',
        message: 'success',
        statusCode: 200,
      };
    } catch (error) {
      throw new Error('Error creating payment');
    }
  }
).use(
  makeHandlerIdempotent({
    persistenceStore,
    config,
  })
);

When enabled, the default is to cache a maximum of 256 records in each Lambda execution environment - You can change it with the maxLocalCacheSize parameter.

Expiring idempotency records¶

By default, we expire idempotency records after an hour (3600 seconds).

In most cases, it is not desirable to store the idempotency records forever. Rather, you want to guarantee that the same payload won't be executed within a period of time.

You can change this window with the expiresAfterSeconds parameter:

Adjusting idempotency record expiration

import {
  IdempotencyConfig,
  makeIdempotent,
} from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import type { Context } from 'aws-lambda';
import type { Request, Response } from './types.js';

const persistenceStore = new DynamoDBPersistenceLayer({
  tableName: 'idempotencyTableName',
});

const config = new IdempotencyConfig({
  expiresAfterSeconds: 300,
});

export const handler = makeIdempotent(
  async (_event: Request, _context: Context): Promise<Response> => {
    try {
      // ... create payment

      return {
        paymentId: '12345',
        message: 'success',
        statusCode: 200,
      };
    } catch (error) {
      throw new Error('Error creating payment');
    }
  },
  {
    persistenceStore,
    config,
  }
);

This will mark any records older than 5 minutes as expired, and your function will be executed as normal if it is invoked with a matching payload.

Idempotency record expiration vs DynamoDB time-to-live (TTL)

DynamoDB TTL is a feature to remove items after a certain period of time, it may occur within 48 hours of expiration.

We don't rely on DynamoDB or any persistence storage layer to determine whether a record is expired to avoid eventual inconsistency states.

Instead, Idempotency records saved in the storage layer contain timestamps that can be verified upon retrieval and double checked within Idempotency feature.

Why?

A record might still be valid (COMPLETE) when we retrieved, but in some rare cases it might expire a second later. A record could also be cached in memory. You might also want to have idempotent transactions that should expire in seconds.

Payload validation¶

Question: What if your function is invoked with the same payload except some outer parameters have changed?

Example: A payment transaction for a given productID was requested twice for the same customer, however the amount to be paid has changed in the second transaction.

By default, we will return the same result as it returned before, however in this instance it may be misleading; we provide a fail fast payload validation to address this edge case.

With payloadValidationJmesPath, you can provide an additional JMESPath expression to specify which part of the event body should be validated against previous idempotent invocations

Payload validation

import { randomUUID } from 'node:crypto';
import {
  IdempotencyConfig,
  makeIdempotent,
} from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import type { Context } from 'aws-lambda';
import type { Request, Response, SubscriptionResult } from './types.js';

const persistenceStore = new DynamoDBPersistenceLayer({
  tableName: 'idempotencyTableName',
});
const config = new IdempotencyConfig({
  eventKeyJmesPath: '["userId", "productId"]',
  payloadValidationJmesPath: 'amount',
});

const fetchProductAmount = async (_transactionId: string): Promise<number> => {
  // ... fetch product amount
  return 42;
};

const createSubscriptionPayment = makeIdempotent(
  async (event: Request & { amount: number }): Promise<SubscriptionResult> => {
    // ... create payment
    return {
      id: randomUUID(),
      productId: event.productId,
    };
  },
  {
    persistenceStore,
    dataIndexArgument: 1,
    config,
  }
);

export const handler = async (
  event: Request,
  context: Context
): Promise<Response> => {
  config.registerLambdaContext(context);
  try {
    const productAmount = await fetchProductAmount(event.productId);
    const payment = await createSubscriptionPayment({
      ...event,
      amount: productAmount,
    });

    return {
      paymentId: payment.id,
      message: 'success',
      statusCode: 200,
    };
  } catch (error) {
    throw new Error('Error creating payment');
  }
};

In this example, the userId and productId keys are used as the payload to generate the idempotency key, as per eventKeyJmespath parameter.

Note

If we try to send the same request but with a different amount, we will raise IdempotencyValidationError.

Without payload validation, we would have returned the same result as we did for the initial request. Since we're also returning an amount in the response, this could be quite confusing for the client.

By using payloadValidationJmesPath="amount", we prevent this potentially confusing behavior and instead throw an error.

Custom JMESPath Functions¶

You can provide custom JMESPath functions for evaluating JMESPath expressions by passing them through the jmesPathOptions parameter. In this example, we use a custom function, my_fancy_function, to parse the payload as a JSON object instead of a string.

Custom JMESPath functions

import type { JSONValue } from '@aws-lambda-powertools/commons/types';
import {
  IdempotencyConfig,
  makeIdempotent,
} from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import {
  Functions,
  PowertoolsFunctions,
} from '@aws-lambda-powertools/jmespath/functions';

const persistenceStore = new DynamoDBPersistenceLayer({
  tableName: 'idempotencyTableName',
});

class MyFancyFunctions extends PowertoolsFunctions {
  @Functions.signature({
    argumentsSpecs: [['string']],
  })
  public funcMyFancyFunction(value: string): JSONValue {
    return JSON.parse(value);
  }
}

export const handler = makeIdempotent(async () => true, {
  persistenceStore,
  config: new IdempotencyConfig({
    eventKeyJmesPath: 'my_fancy_function(body).["user", "productId"]',
    jmesPathOptions: new MyFancyFunctions(),
  }),
});

Making idempotency key required¶

If you want to enforce that an idempotency key is required, you can set throwOnNoIdempotencyKey to true.

This means that we will raise IdempotencyKeyError if the evaluation of eventKeyJmesPath results in an empty subset.

Warning

To prevent errors, transactions will not be treated as idempotent if throwOnNoIdempotencyKey is set to false and the evaluation of eventKeyJmesPath is an empty result. Therefore, no data will be fetched, stored, or deleted in the idempotency storage layer.

Idempotency key requiredSuccess EventFailure Event

import {
  IdempotencyConfig,
  makeIdempotent,
} from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import type { Context } from 'aws-lambda';
import type { Request, Response } from './types.js';

const persistenceStore = new DynamoDBPersistenceLayer({
  tableName: 'idempotencyTableName',
});

const config = new IdempotencyConfig({
  throwOnNoIdempotencyKey: true,
  eventKeyJmesPath: '["user.uid", "productId"]',
});

export const handler = makeIdempotent(
  async (_event: Request, _context: Context): Promise<Response> => {
    try {
      // ... create payment

      return {
        paymentId: '12345',
        message: 'success',
        statusCode: 200,
      };
    } catch (error) {
      throw new Error('Error creating payment');
    }
  },
  {
    persistenceStore,
    config,
  }
);

{
  "user": {
    "uid": "BB0D045C-8878-40C8-889E-38B3CB0A61B1",
    "name": "Foo"
  },
  "productId": 10000
}

{
  "user": {
    "uid": "BB0D045C-8878-40C8-889E-38B3CB0A61B1",
    "name": "foo",
    "productId": 10000
  }
}

Batch integration¶

You can easily integrate with Batch utility by using idempotency wrapper around your processing function. This ensures that you process each record in an idempotent manner, and guard against a Lambda timeout idempotent situation.

Choosing a unique batch record attribute

In this example, we choose messageId as our idempotency key since we know it'll be unique. Depending on your use case, it might be more accurate to choose another field your producer intentionally set to define uniqueness.

Integration with batch processorSample event

import {
  BatchProcessor,
  EventType,
  processPartialResponse,
} from '@aws-lambda-powertools/batch';
import {
  IdempotencyConfig,
  makeIdempotent,
} from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import type {
  Context,
  SQSBatchResponse,
  SQSEvent,
  SQSRecord,
} from 'aws-lambda';

const processor = new BatchProcessor(EventType.SQS);

const dynamoDBPersistence = new DynamoDBPersistenceLayer({
  tableName: 'idempotencyTable',
});
const idempotencyConfig = new IdempotencyConfig({
  eventKeyJmesPath: 'messageId',
});

const processIdempotently = makeIdempotent(
  async (_record: SQSRecord) => {
    // process your event
  },
  {
    persistenceStore: dynamoDBPersistence,
    config: idempotencyConfig,
  }
);

export const handler = async (
  event: SQSEvent,
  context: Context
): Promise<SQSBatchResponse> => {
  idempotencyConfig.registerLambdaContext(context);

  return processPartialResponse(event, processIdempotently, processor, {
    context,
  });
};

{
  "Records": [
    {
      "messageId": "059f36b4-87a3-44ab-83d2-661975830a7d",
      "receiptHandle": "AQEBwJnKyrHigUMZj6rYigCgxlaS3SLy0a...",
      "body": "Test message.",
      "attributes": {
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1545082649183",
        "SenderId": "AIDAIENQZJOLO23YVJ4VO",
        "ApproximateFirstReceiveTimestamp": "1545082649185"
      },
      "messageAttributes": {
        "testAttr": {
          "stringValue": "100",
          "binaryValue": "base64Str",
          "dataType": "Number"
        }
      },
      "md5OfBody": "e4e68fb7bd0e697a0ae8f1bb342846b3",
      "eventSource": "aws:sqs",
      "eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:my-queue",
      "awsRegion": "us-east-2"
    }
  ]
}

Customizing AWS SDK configuration¶

The clientConfig and awsSdkV3Client parameters enable you to pass in custom configurations or your own DynamoDBClient when constructing the persistence store.

Passing specific configurationPassing custom DynamoDBClient

import { makeIdempotent } from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import type { Context } from 'aws-lambda';
import type { Request, Response } from './types.js';

const persistenceStore = new DynamoDBPersistenceLayer({
  tableName: 'idempotencyTableName',
  clientConfig: {
    region: 'us-east-1',
  },
});

export const handler = makeIdempotent(
  async (_event: Request, _context: Context): Promise<Response> => {
    try {
      // ... create payment

      return {
        paymentId: '12345',
        message: 'success',
        statusCode: 200,
      };
    } catch (error) {
      throw new Error('Error creating payment');
    }
  },
  {
    persistenceStore,
  }
);

import { makeIdempotent } from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import { DynamoDBClient } from '@aws-sdk/client-dynamodb';
import type { Context } from 'aws-lambda';
import type { Request, Response } from './types.js';

const customDynamoDBClient = new DynamoDBClient({
  endpoint: 'http://localhost:8000',
});
const persistenceStore = new DynamoDBPersistenceLayer({
  tableName: 'idempotencyTableName',
  awsSdkV3Client: customDynamoDBClient,
});

export const handler = makeIdempotent(
  async (_event: Request, _context: Context): Promise<Response> => {
    try {
      // ... create payment

      return {
        paymentId: '12345',
        message: 'success',
        statusCode: 200,
      };
    } catch (error) {
      throw new Error('Error creating payment');
    }
  },
  {
    persistenceStore,
  }
);

Using a DynamoDB table with a composite primary key¶

When using a composite primary key table (hash+range key), use sortKeyAttr parameter when initializing your persistence layer.

With this setting, we will save the idempotency key in the sort key instead of the primary key. By default, the primary key will now be set to idempotency#{LAMBDA_FUNCTION_NAME}.

You can optionally set a static value for the partition key using the staticPkValue parameter.

Reusing a DynamoDB table that uses a composite primary key

import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import { makeHandlerIdempotent } from '@aws-lambda-powertools/idempotency/middleware';
import middy from '@middy/core';
import type { Context } from 'aws-lambda';
import type { Request, Response } from './types.js';

const persistenceStore = new DynamoDBPersistenceLayer({
  tableName: 'idempotencyTableName',
  sortKeyAttr: 'sort_key',
});

export const handler = middy(
  async (_event: Request, _context: Context): Promise<Response> => {
    try {
      // ... create payment

      return {
        paymentId: '12345',
        message: 'success',
        statusCode: 200,
      };
    } catch (error) {
      throw new Error('Error creating payment');
    }
  }
).use(
  makeHandlerIdempotent({
    persistenceStore,
  })
);

The example function above would cause data to be stored in DynamoDB like this:

id	sort_key	expiration	status	data
idempotency#MyLambdaFunction	1e956ef7da78d0cb890be999aecc0c9e	1636549553	COMPLETED	{"paymentId": "12345, "message": "success", "statusCode": 200}
idempotency#MyLambdaFunction	2b2cdb5f86361e97b4383087c1ffdf27	1636549571	COMPLETED	{"paymentId": "527212", "message": "success", "statusCode": 200}
idempotency#MyLambdaFunction	f091d2527ad1c78f05d54cc3f363be80	1636549585	IN_PROGRESS

Bring your own persistent store¶

This utility provides an abstract base class (ABC), so that you can implement your choice of persistent storage layer.

You can create your own persistent store from scratch by inheriting the BasePersistenceLayer class, and implementing _getRecord(), _putRecord(), _updateRecord() and _deleteRecord().

_getRecord() – Retrieves an item from the persistence store using an idempotency key and returns it as a IdempotencyRecord instance.
_putRecord() – Adds a IdempotencyRecord to the persistence store if it doesn't already exist with that key. Throws an IdempotencyItemAlreadyExistsError error if a non-expired entry already exists.
_updateRecord() – Updates an item in the persistence store.
_deleteRecord() – Removes an item from the persistence store.

Below an example implementation of a custom persistence layer backed by a generic key-value store.

CustomPersistenceLayerindex.tstypes.ts

import {
  IdempotencyItemAlreadyExistsError,
  IdempotencyItemNotFoundError,
  IdempotencyRecordStatus,
} from '@aws-lambda-powertools/idempotency';
import {
  BasePersistenceLayer,
  IdempotencyRecord,
} from '@aws-lambda-powertools/idempotency/persistence';
import type { IdempotencyRecordOptions } from '@aws-lambda-powertools/idempotency/types';
import { Transform } from '@aws-lambda-powertools/parameters';
import { getSecret } from '@aws-lambda-powertools/parameters/secrets';
import {
  ProviderClient,
  ProviderItemAlreadyExists,
} from './advancedBringYourOwnPersistenceLayerProvider';
import type { ApiSecret, ProviderItem } from './types';

class CustomPersistenceLayer extends BasePersistenceLayer {
  #collectionName: string;
  #client?: ProviderClient;

  public constructor(config: { collectionName: string }) {
    super();
    this.#collectionName = config.collectionName;
  }

  protected async _deleteRecord(record: IdempotencyRecord): Promise<void> {
    await (await this.#getClient()).delete(
      this.#collectionName,
      record.idempotencyKey
    );
  }

  protected async _getRecord(
    idempotencyKey: string
  ): Promise<IdempotencyRecord> {
    try {
      const item = await (await this.#getClient()).get(
        this.#collectionName,
        idempotencyKey
      );

      return new IdempotencyRecord({
        ...(item as unknown as IdempotencyRecordOptions),
      });
    } catch (error) {
      throw new IdempotencyItemNotFoundError();
    }
  }

  protected async _putRecord(record: IdempotencyRecord): Promise<void> {
    const item: Partial<ProviderItem> = {
      status: record.getStatus(),
    };

    if (record.inProgressExpiryTimestamp !== undefined) {
      item.in_progress_expiration = record.inProgressExpiryTimestamp;
    }

    if (this.isPayloadValidationEnabled() && record.payloadHash !== undefined) {
      item.validation = record.payloadHash;
    }

    const ttl = record.expiryTimestamp
      ? Math.floor(new Date(record.expiryTimestamp * 1000).getTime() / 1000) -
        Math.floor(new Date().getTime() / 1000)
      : this.getExpiresAfterSeconds();

    let existingItem: ProviderItem | undefined;
    try {
      existingItem = await (await this.#getClient()).put(
        this.#collectionName,
        record.idempotencyKey,
        item,
        {
          ttl,
        }
      );
    } catch (error) {
      if (error instanceof ProviderItemAlreadyExists) {
        if (
          existingItem &&
          existingItem.status !== IdempotencyRecordStatus.INPROGRESS &&
          (existingItem.in_progress_expiration || 0) < Date.now()
        ) {
          throw new IdempotencyItemAlreadyExistsError(
            `Failed to put record for already existing idempotency key: ${record.idempotencyKey}`
          );
        }
      }
    }
  }

  protected async _updateRecord(record: IdempotencyRecord): Promise<void> {
    const value: Partial<ProviderItem> = {
      data: JSON.stringify(record.responseData),
      status: record.getStatus(),
    };

    if (this.isPayloadValidationEnabled()) {
      value.validation = record.payloadHash;
    }

    await (await this.#getClient()).update(
      this.#collectionName,
      record.idempotencyKey,
      value
    );
  }

  async #getClient(): Promise<ProviderClient> {
    if (this.#client) return this.#client;

    const secretName = process.env.API_SECRET;
    if (!secretName) {
      throw new Error('API_SECRET environment variable is not set');
    }

    const apiSecret = await getSecret<ApiSecret>(secretName, {
      transform: Transform.JSON,
    });

    if (!apiSecret) {
      throw new Error(`Could not retrieve secret ${secretName}`);
    }

    this.#client = new ProviderClient({
      apiKey: apiSecret.apiKey,
      defaultTtlSeconds: this.getExpiresAfterSeconds(),
    });

    return this.#client;
  }
}

export { CustomPersistenceLayer };

import { randomUUID } from 'node:crypto';
import {
  IdempotencyConfig,
  makeIdempotent,
} from '@aws-lambda-powertools/idempotency';
import type { Context } from 'aws-lambda';
import { CustomPersistenceLayer } from './advancedBringYourOwnPersistenceLayer';
import type { Request, Response, SubscriptionResult } from './types.js';

const persistenceStore = new CustomPersistenceLayer({
  collectionName: 'powertools',
});
const config = new IdempotencyConfig({
  expiresAfterSeconds: 60,
});

const createSubscriptionPayment = makeIdempotent(
  async (
    _transactionId: string,
    event: Request
  ): Promise<SubscriptionResult> => {
    // ... create payment
    return {
      id: randomUUID(),
      productId: event.productId,
    };
  },
  {
    persistenceStore,
    dataIndexArgument: 1,
    config,
  }
);

export const handler = async (
  event: Request,
  context: Context
): Promise<Response> => {
  config.registerLambdaContext(context);

  try {
    const transactionId = randomUUID();
    const payment = await createSubscriptionPayment(transactionId, event);

    return {
      paymentId: payment.id,
      message: 'success',
      statusCode: 200,
    };
  } catch (error) {
    throw new Error('Error creating payment');
  }
};

import type { IdempotencyRecordStatusValue } from '@aws-lambda-powertools/idempotency/types';

export type Request = {
  user: string;
  productId: string;
};

export type Response = {
  [key: string]: unknown;
};

export type SubscriptionResult = {
  id: string;
  productId: string;
};

export type ApiSecret = {
  apiKey: string;
  refreshToken: string;
  validUntil: number;
  restEndpoint: string;
};

export type ProviderItem = {
  validation?: string;
  in_progress_expiration?: number;
  status: IdempotencyRecordStatusValue;
  data: string;
};

Danger

Pay attention to the documentation for each - you may need to perform additional checks inside these methods to ensure the idempotency guarantees remain intact.

For example, the _putRecord() method needs to throw an error if a non-expired record already exists in the data store with a matching key.

Manipulating the Idempotent Response¶

You can set up a responseHook in the IdempotentConfig class to manipulate the returned data when an operation is idempotent. The hook function will be called with the current deserialized response object and the Idempotency record.

Using an Idempotent Response HookSample eventSample Idempotent response

import { randomUUID } from 'node:crypto';
import type { JSONValue } from '@aws-lambda-powertools/commons/types';
import {
  IdempotencyConfig,
  makeIdempotent,
} from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import type { IdempotencyRecord } from '@aws-lambda-powertools/idempotency/persistence';
import type { Context } from 'aws-lambda';
import type { Request, Response, SubscriptionResult } from './types.js';

const persistenceStore = new DynamoDBPersistenceLayer({
  tableName: 'idempotencyTableName',
});

const responseHook = (response: JSONValue, record: IdempotencyRecord) => {
  // Return inserted Header data into the Idempotent Response
  (response as Response).headers = {
    'x-idempotency-key': record.idempotencyKey,
  };

  // Must return the response here
  return response as JSONValue;
};

const config = new IdempotencyConfig({
  responseHook,
});

const createSubscriptionPayment = async (
  event: Request
): Promise<SubscriptionResult> => {
  // ... create payment
  return {
    id: randomUUID(),
    productId: event.productId,
  };
};

export const handler = makeIdempotent(
  async (event: Request, _context: Context): Promise<Response> => {
    try {
      const payment = await createSubscriptionPayment(event);

      return {
        paymentId: payment.id,
        message: 'success',
        statusCode: 200,
      };
    } catch (error) {
      throw new Error('Error creating payment');
    }
  },
  {
    persistenceStore,
    config,
  }
);

{
    "user": "John Doe",
    "productId": "123456"
}

{
    "message": "success",
    "paymentId": "31a964eb-7477-4fe1-99fe-7f8a6a351a7e",
    "statusCode": 200,
    "headers": {
      "x-idempotency-key": "function-name#mHfGv2vJ8h+ZvLIr/qGBbQ=="
    }
  }

Info: Using custom de-serialization?

The responseHook is called after the custom de-serialization so the payload you process will be the de-serialized version.

Being a good citizen¶

When using response hooks to manipulate returned data from idempotent operations, it's important to follow best practices to avoid introducing complexity or issues. Keep these guidelines in mind:

Response hook works exclusively when operations are idempotent. The hook will not be called when an operation is not idempotent, or when the idempotent logic fails.
Catch and Handle Exceptions. Your response hook code should catch and handle any exceptions that may arise from your logic. Unhandled exceptions will cause the Lambda function to fail unexpectedly.
Keep Hook Logic Simple Response hooks should consist of minimal and straightforward logic for manipulating response data. Avoid complex conditional branching and aim for hooks that are easy to reason about.

Testing your code¶

The idempotency utility provides several routes to test your code.

Disabling the idempotency utility¶

When testing your code, you may wish to disable the idempotency logic altogether and focus on testing your business logic. To do this, you can set the environment variable POWERTOOLS_IDEMPOTENCY_DISABLED with a truthy value.

Testing with local DynamoDB¶

When testing your Lambda function locally, you can use a local DynamoDB instance to test the idempotency feature. You can use DynamoDB Local or LocalStack.

handler.test.tshandler.ts

import { makeIdempotent } from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import type { Context } from 'aws-lambda';
import { handler } from './workingWithLocalDynamoDB';

describe('Idempotent handler', () => {
  const lambdaContext = {
    functionName: 'foo-bar-function',
    memoryLimitInMB: '128',
    invokedFunctionArn:
      'arn:aws:lambda:eu-west-1:123456789012:function:foo-bar-function',
    awsRequestId: 'c6af9ac6-7b61-11e6-9a41-93e812345678',
    getRemainingTimeInMillis: () => 1234,
  } as Context;

  const mockPersistenceStore = new DynamoDBPersistenceLayer({
    tableName: 'IdempotencyTable',
    clientConfig: { endpoint: 'http://localhost:8000' }, // 8000 for local DynamoDB and 4566 for LocalStack
  });

  const idempotentHandler = makeIdempotent(handler, {
    persistenceStore: mockPersistenceStore,
  });

  it('should return the same response', async () => {
    const response = await idempotentHandler(
      {
        foo: 'bar',
      },
      lambdaContext
    );
    expect(response).toEqual({
      statusCode: 200,
      body: JSON.stringify({
        paymentId: '123',
        message: 'Payment created',
      }),
    });
  });
});

import { makeIdempotent } from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import type { Context } from 'aws-lambda';

const ddbPersistenceStore = new DynamoDBPersistenceLayer({
  tableName: 'IdempotencyTable',
});

const handler = async (event: unknown, context: Context) => {
  return {
    statusCode: 200,
    body: JSON.stringify({
      paymentId: '123',
      message: 'Payment created',
    }),
  };
};

const idempotentHandler = makeIdempotent(handler, {
  persistenceStore: ddbPersistenceStore,
});

export { idempotentHandler, handler };

Extra resources¶

If you're interested in a deep dive on how Amazon uses idempotency when building our APIs, check out this article.