Skip to content

JMESPath Functions

Built-in JMESPath functions to easily deserialize common encoded JSON payloads in Lambda functions.

Key features

  • Deserialize JSON from JSON strings, base64, and compressed data
  • Use JMESPath to extract and combine data recursively
  • Provides commonly used JMESPath expressions with popular event sources

Getting started

You might have events that contain encoded JSON payloads as string, base64, or even in compressed format. It is a common use case to decode and extract them partially or fully as part of your Lambda function invocation.

Powertools for AWS Lambda (TypeScript) also have utilities like idempotency where you might need to extract a portion of your data before using them.

Terminology

Envelope is the terminology we use for the JMESPath expression to extract your JSON object from your data input. We might use those two terms interchangeably.

Extracting data

You can use the extractDataFromEnvelope function with any JMESPath expression.

Tip

Another common use case is to fetch deeply nested data, filter, flatten, and more.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
import { extractDataFromEnvelope } from '@aws-lambda-powertools/jmespath/envelopes';

type MyEvent = {
  body: string; // "{\"customerId\":\"dd4649e6-2484-4993-acb8-0f9123103394\"}"
  deeplyNested: Array<{ someData: number[] }>;
};

type MessageBody = {
  customerId: string;
};

export const handler = async (event: MyEvent): Promise<unknown> => {
  const payload = extractDataFromEnvelope<MessageBody>(
    event,
    'powertools_json(body)'
  );
  const { customerId } = payload; // now deserialized

  // also works for fetching and flattening deeply nested data
  const someData = extractDataFromEnvelope<number[]>(
    event,
    'deeplyNested[*].someData[]'
  );

  return {
    customerId,
    message: 'success',
    context: someData,
    statusCode: 200,
  };
};

1
2
3
4
5
6
7
8
{
  "body": "{\"customerId\":\"dd4649e6-2484-4993-acb8-0f9123103394\"}",
  "deeplyNested": [
    {
      "someData": [1, 2, 3]
    }
  ]
}

Built-in envelopes

We provide built-in envelopes for popular AWS Lambda event sources to easily decode and/or deserialize JSON objects.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
import {
  extractDataFromEnvelope,
  SQS,
} from '@aws-lambda-powertools/jmespath/envelopes';
import { Logger } from '@aws-lambda-powertools/logger';
import type { SQSEvent } from 'aws-lambda';

const logger = new Logger();

type MessageBody = {
  customerId: string;
};

export const handler = async (event: SQSEvent): Promise<void> => {
  const records = extractDataFromEnvelope<Array<MessageBody>>(event, SQS);
  for (const record of records) {
    // records is now a list containing the deserialized body of each message
    const { customerId } = record;
    logger.appendKeys({ customerId });
  }
};

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
{
  "Records": [
    {
      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
      "receiptHandle": "MessageReceiptHandle",
      "body": "{\"customerId\":\"dd4649e6-2484-4993-acb8-0f9123103394\",\"booking\":{\"id\":\"5b2c4803-330b-42b7-811a-c68689425de1\",\"reference\":\"ySz7oA\",\"outboundFlightId\":\"20c0d2f2-56a3-4068-bf20-ff7703db552d\"},\"payment\":{\"receipt\":\"https://pay.stripe.com/receipts/acct_1Dvn7pF4aIiftV70/ch_3JTC14F4aIiftV700iFq2CHB/rcpt_K7QsrFln9FgFnzUuBIiNdkkRYGxUL0X\",\"amount\":100}}",
      "attributes": {
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000",
        "SenderId": "123456789012",
        "ApproximateFirstReceiveTimestamp": "1523232000001"
      },
      "messageAttributes": {},
      "md5OfBody": "7b270e59b47ff90a553787216d55d91d",
      "eventSource": "aws:sqs",
      "eventSourceARN": "arn:aws:sqs:us-east-1:123456789012:MyQueue",
      "awsRegion": "us-east-1"
    }
  ]
}

These are all built-in envelopes you can use along with their expression as a reference:

Envelope JMESPath expression
API_GATEWAY_HTTP powertools_json(body)
API_GATEWAY_REST powertools_json(body)
CLOUDWATCH_EVENTS_SCHEDULED detail
CLOUDWATCH_LOGS awslogs.powertools_base64_gzip(data) | powertools_json(@).logEvents[*]
EVENTBRIDGE detail
KINESIS_DATA_STREAM Records[*].kinesis.powertools_json(powertools_base64(data))
S3_EVENTBRIDGE_SQS Records[*].powertools_json(body).detail
S3_KINESIS_FIREHOSE records[*].powertools_json(powertools_base64(data)).Records[0]
S3_SNS_KINESIS_FIREHOSE records[*].powertools_json(powertools_base64(data)).powertools_json(Message).Records[0]
S3_SNS_SQS Records[*].powertools_json(body).powertools_json(Message).Records[0]
S3_SQS Records[*].powertools_json(body).Records[0]
SNS Records[0].Sns.Message | powertools_json(@)
SQS Records[*].powertools_json(body)
Using SNS?

If you don't require SNS metadata, enable raw message delivery. It will reduce multiple payload layers and size, when using SNS in combination with other services (e.g., SQS, S3, etc).

Advanced

Built-in JMESPath functions

You can use our built-in JMESPath functions within your envelope expression. They handle deserialization for common data formats found in AWS Lambda event sources such as JSON strings, base64, and uncompress gzip data.

powertools_json function

Use powertools_json function to decode any JSON string anywhere a JMESPath expression is allowed.

Idempotency scenario

This sample will deserialize the JSON string within the body key before Idempotency processes it.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
import {
  IdempotencyConfig,
  makeIdempotent,
} from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import type { APIGatewayEvent } from 'aws-lambda';
import { randomUUID } from 'node:crypto';

const persistenceStore = new DynamoDBPersistenceLayer({
  tableName: 'IdempotencyTable',
});

export const handler = makeIdempotent(
  async (event: APIGatewayEvent) => {
    const body = JSON.parse(event.body || '{}');
    const { user, productId } = body;

    const result = await createSubscriptionPayment(user, productId);

    return {
      statusCode: 200,
      body: JSON.stringify({
        paymentId: result.id,
        message: 'success',
      }),
    };
  },
  {
    persistenceStore,
    config: new IdempotencyConfig({
      eventKeyJmesPath: 'powertools_json(body)',
    }),
  }
);

const createSubscriptionPayment = async (
  user: string,
  productId: string
): Promise<{ id: string; message: string }> => {
  const payload = { user, productId };
  const response = await fetch('https://httpbin.org/anything', {
    method: 'POST',
    body: JSON.stringify(payload),
  });

  if (!response.ok) {
    throw new Error('Failed to create subscription payment');
  }

  return { id: randomUUID(), message: 'paid' };
};

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
{
  "version": "2.0",
  "routeKey": "ANY /createpayment",
  "rawPath": "/createpayment",
  "rawQueryString": "",
  "headers": {
    "Header1": "value1",
    "Header2": "value2"
  },
  "requestContext": {
    "accountId": "123456789012",
    "apiId": "api-id",
    "domainName": "id.execute-api.us-east-1.amazonaws.com",
    "domainPrefix": "id",
    "http": {
      "method": "POST",
      "path": "/createpayment",
      "protocol": "HTTP/1.1",
      "sourceIp": "ip",
      "userAgent": "agent"
    },
    "requestId": "id",
    "routeKey": "ANY /createpayment",
    "stage": "$default",
    "time": "10/Feb/2021:13:40:43 +0000",
    "timeEpoch": 1612964443723
  },
  "body": "{\"user\":\"xyz\",\"product_id\":\"123456789\"}",
  "isBase64Encoded": false
}

powertools_base64 function

Use powertools_base64 function to decode any base64 data.

This sample will decode the base64 value within the data key, and deserialize the JSON string before processing.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
import { extractDataFromEnvelope } from '@aws-lambda-powertools/jmespath/envelopes';
import { Logger } from '@aws-lambda-powertools/logger';

const logger = new Logger();

export const handler = async (event: { payload: string }): Promise<void> => {
  const data = extractDataFromEnvelope<string>(
    event,
    'powertools_json(powertools_base64(payload))'
  );

  logger.info('Decoded payload', { data }); // (1)!
};
  1. The data variable contains the decoded object that looks like this: 
    1
2
3
4
5
6
7
    {
    user_id: 123,
    product_id: 1,
    quantity: 2,
    price: 10.4,
    currency: 'USD',
}

1
2
3
{
  "payload": "eyJ1c2VyX2lkIjogMTIzLCAicHJvZHVjdF9pZCI6IDEsICJxdWFudGl0eSI6IDIsICJwcmljZSI6IDEwLjQwLCAiY3VycmVuY3kiOiAiVVNEIn0="
}

powertools_base64_gzip function

Use powertools_base64_gzip function to decompress and decode base64 data.

This sample will decompress and decode base64 data from Cloudwatch Logs, then use JMESPath pipeline expression to pass the result for decoding its JSON string.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
import { extractDataFromEnvelope } from '@aws-lambda-powertools/jmespath/envelopes';
import { Logger } from '@aws-lambda-powertools/logger';

const logger = new Logger();

export const handler = async (event: { payload: string }): Promise<void> => {
  const logGroup = extractDataFromEnvelope<string>(
    event, // (1)!
    'powertools_base64_gzip(payload) | powertools_json(@).logGroup'
  );

  logger.info('Log group name', { logGroup }); // (2)!
};
  1. The payload key contains a JSON object that once decompressed and decoded looks like this: 
     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
    {
    "owner": "123456789012",
    "logGroup": "/aws/lambda/powertools-example",
    "logStream": "2020/09/02/[$LATEST]d3a8dcaffc7f4de2b8db132e3e106660",
    "subscriptionFilters": ["Destination"],
    "messageType": "DATA_MESSAGE",
    "logEvents": [
        {
            "id": "eventId1",
            "message": {
                "username": "lessa",
                "message": "hello world"
            },
            "timestamp": 1440442987000
        },
        {
            "id": "eventId2",
            "message": {
                "username": "dummy",
                "message": "hello world"
            },
            "timestamp": 1440442987001
        }
    ]
}
  2. The logGroup variable contains the string "/aws/lambda/powertools-example".
1
2
3
{
  "payload": "H4sIACZAXl8C/52PzUrEMBhFX2UILpX8tPbHXWHqIOiq3Q1F0ubrWEiakqTWofTdTYYB0YWL2d5zvnuTFellBIOedoiyKH5M0iwnlKH7HZL6dDB6ngLDfLFYctUKjie9gHFaS/sAX1xNEq525QxwFXRGGMEkx4Th491rUZdV3YiIZ6Ljfd+lfSyAtZloacQgAkqSJCGhxM6t7cwwuUGPz4N0YKyvO6I9WDeMPMSo8Z4Ca/kJ6vMEYW5f1MX7W1lVxaG8vqX8hNFdjlc0iCBBSF4ERT/3Pl7RbMGMXF2KZMh/C+gDpNS7RRsp0OaRGzx0/t8e0jgmcczyLCWEePhni/23JWalzjdu0a3ZvgEaNLXeugEAAA=="
}

Bring your own JMESPath function

Warning

This should only be used for advanced use cases where you have special formats not covered by the built-in functions.

For special binary formats that you want to decode before processing, you can bring your own JMESPath function by extending the PowertoolsFunctions class.

Here is an example of how to decompress messages compressed using the Brotli compression algorithm:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
 import { fromBase64 } from '@aws-lambda-powertools/commons/utils/base64';
 import { extractDataFromEnvelope } from '@aws-lambda-powertools/jmespath/envelopes';
 import { PowertoolsFunctions } from '@aws-lambda-powertools/jmespath/functions';
 import { Logger } from '@aws-lambda-powertools/logger';
 import { brotliDecompressSync } from 'node:zlib';

 const logger = new Logger();

 class CustomFunctions extends PowertoolsFunctions {
   @PowertoolsFunctions.signature({ // (1)!
     argumentsSpecs: [['string']],
     variadic: false,
   })
   public funcDecodeBrotliCompression(value: string): string { // (2)!
     const encoded = fromBase64(value, 'base64');
     const uncompressed = brotliDecompressSync(encoded);

     return uncompressed.toString();
   }
 }

 export const handler = async (event: { payload: string }): Promise<void> => {
   const message = extractDataFromEnvelope<string>(
     event,
     'Records[*].decode_brotli_compression(notification) | [*].powertools_json(@).message',
     { customFunctions: new CustomFunctions() }
   );

   logger.info('Decoded message', { message });
 };
  1. The function signature can be enforced at runtime by using the @Functions.signature decorator.
  2. The name of the function must start with the func prefix.
1
2
3
4
5
6
7
8
9
{
  "Records": [
    {
      "application": "app",
      "datetime": "2022-01-01T00:00:00.000Z",
      "notification": "GyYA+AXhZKk/K5DkanoQSTYpSKMwwxXh8DRWVo9A1hLqAQ=="
    }
  ]
}