How to Subscribe Private Endpoint Internal HTTP URL to SNS - AWS

Updated on:

In this article, we will see an interesting use case or system design where we want to connect AWS SNS to an internal HTTP or HTTPS service.

SNS is a public service designed to connect to external public HTTPS endpoints.

SNS is purpose-built for external communications and it supports Public HTTP/HTTPS endpoints to subscribe directly

What if you want to add an internal HTTP endpoint from your VPC as a subscription -  you need to workaround with Proxies

Event Bridge Header

There are two types of Proxies we can use.

  1. Lambda Proxy -  Using Lambda Function as a Proxy
  2. Event Bridge and API Gateway Service Proxy - Using Event Bridge Pipe and API Gateway as a Proxy

 

Lambda Proxy

In this model, a Lambda function acts as a Proxy to forward the calls to internal services and HTTP endpoints

In the following diagram, you can see that Lambda is added as a subscription to the SNS

Here, Lambda does three things

Public to Private Conversion

The first and Primary responsibility of this pattern - is to convert the Public connection to a private connection and thereby able to connect to all the internal Endpoints and URLs  - This Service Proxy Lambda would be inside the VPC and run on the private Subnets and can DNS resolve internal Route 53 entries

Enrichment ( Validation & Transformation) 

In this stage - Lambda can optionally parse the SNS payload. Validate and transform it to another schema accepted by the destination service.

Adding additional headers or inputs based on the Source or event_type attributes, which might be needed by the destination service for authentication or core functionality

Logging and Observability can also be enabled along with data sanitization if need be

But this is optional, If we want to forward the SNS message to the HTTP service as is, this can be skipped.

Authentication

This may be optional too - but in a few cases, we might want to add some Authorization token or header before passing the request to the internal service.

We might want to take some info from the Passed payload from SNS and forward them as headers like X-API-Key  or X-Origin-Service for authentication

Here is a sample Lambda Proxy that does all three things.

import json
import os
import re
import requests

def lambda_handler(event, context):
    try:
        # Read SNS payload
        sns_message = event['Records'][0]['Sns']['Message']
        payload = json.loads(sns_message)

        # Enrich: Validate JSON schema
        if not validate_payload(payload):
            raise ValueError("Invalid payload schema")

        # Authenticate: Extract headers and remove them from payload
        headers = {
            'X-User-Email': payload.pop('X-User-Email', None),
            'X-User-Token': payload.pop('X-User-Token', None)
        }
        if None in headers.values():
            raise ValueError("Authentication headers missing in payload")

        # Route: Determine the target service based on event_type
        event_type = payload.get('event_type')
        target_url = get_target_url(event_type)
        if not target_url:
            raise ValueError("Invalid event_type or target service not defined")

        # Forward the request to the target service
        response = requests.post(target_url, headers=headers, json=payload)

        return {
            'statusCode': response.status_code,
            'body': response.text
        }
    except Exception as e:
        return {
            'statusCode': 500,
            'body': str(e)
        }

def validate_payload(payload):
    """
    Validates the payload schema. Modify this function based on your schema requirements.
    """
    required_fields = {'event_type', 'X-User-Email', 'X-User-Token'}
    if not required_fields.issubset(payload.keys()):
        return False

    if not re.match(r"[^@]+@[^@]+\.[^@]+", payload['X-User-Email']):
        return False

    return True

def get_target_url(event_type):
    """
    Determines the target service URL based on event_type.
    """
    service_urls = {
        'serviceA': os.getenv('serviceA'),
        'serviceB': os.getenv('serviceB'),
        'serviceC': os.getenv('serviceC')
    }
    return service_urls.get(event_type)

# Sample event structure for testing locally
if __name__ == "__main__":
    sample_event = {
        "Records": [
            {
                "Sns": {
                    "Message": json.dumps({
                        "event_type": "serviceA",
                        "X-User-Email": "[email protected]",
                        "X-User-Token": "token123",
                        "data": "sample data"
                    })
                }
            }
        ]
    }
    print(lambda_handler(sample_event, None))

This is one of the patterns where Lambda acts as a Proxy and with Lambda being serverless - this architecture can scale to meet growing demands in future and handle higher concurrency too.

But Lambda has a few concerns like Cold start, Concurrency limit and Dynamic Pricing based on resource consumption etc.

Let us explore the other solution now.

 

Event Bridge & API Gateway Service Proxy

In this alternate approach. Instead of Lambda, we are going to use API Gateway as a Proxy.

While you can use the Public API Gateway as an HTTPS subscription directly with SNS - One Problem might arise with Authentication.

SNS does not support Authenticated HTTP/HTTPS endpoints - There is no provisioning to define the Authentication Headers or Keys during the creation of Subscription

SNS HTTPS Subscription

So we need some additional wrapper service that enables us to Authenticate with the API Gateway

That's where we bring the Event Bridge Pipe and SQS.

SNS cannot directly communicate with the API Gateway or Event Bridge either - ( There is firehose but that is a different implementation) so we need SQS here as a Connector

Refer to the following System Design architecture with SQS and EventBridge

SNS HTTP Event Bridge

Steps Involved

To achieve this system design the following steps are involved.

  1. Create an SQS queue and use it for the SNS subscription
  2. In the SQS, Create an Event Bridge Pipe
  3. In Event Bridge we can do four things
    1. Filter - In case we want to filter out certain messages
    2. Enrich - For Payload Transformation - Essential for our current requirement as we need to read the payload get the data and use it for Authentication
    3. Authenticate - You can define what are the headers to be sent to the API Gateway
    4. Target - Define API gateway as the Target ( as of now only public API gateways are supported in Event Bridge)
  4. Create the API Gateway with a Swagger that acts as a Service Proxy
  5. Create VPC links and connect to NLBs inside our VPC - In our case, these NLBs are directly bound with the EKS service

We will not cover all the steps, but I will provide the details and configuration elements required to implement this setup.

The Core elements of this design are Event Bridge Pipe and API Gateway and how to configure the Parsing and Authentication lets explore it a little deeper.

Creating an SNS Subscription, creating a VPC Link inside the API gateway, and creating NLBs via Kubernetes all require some prerequisite knowledge of the respective areas.

You might be already aware - If not, Please refer to the following links

I hope you have the SNS, SQS and the NLB along with VPC links required before proceeding further.

Now let us switch to our Objective on learning how to use the internal NLB service as a Subscription with SNS

Before we configure the Event Bridge - we need an API gateway as a prerequisite so let's cover that first

 

API Gateway Configuration

Our API Gateway Setup is going to consist of the following parts

  1. Authorizer - A Lambda function, written in Python to Authorize the request based on the presence of specific headers or query string - in our case its headers
  2. Swagger File - Open API Specification Swagger File for Creating the REST API contains the API endpoints and methods and Authorizer reference.

Following is the Technical diagram of how our API Gateway looks and functions.

API Gateway configuration

You can download the Swagger file and the Python Authorizer Lambda function code from the following Github Repository link

https://github.com/AKSarav/SampleAPIGateway

As you clone this repository you can find the Authorizer Source Code and API Gateway Swagger file in their respective directories.

In VS Code - you can use the Swagger Viewer Plugin  to view the Swagger file in a nice API documentation format

This is how it looks on the VS Code with Swagger Viewer

You must replace the Endpoints and Methods, Parameters and The Authorizer ARN inside the Swagger file before using it. This is just a sample swagger for reference

Once you have created the Service Proxy API gateway and the Authorizer Lambda and configured the VPC links and NLBs. you are good to go forward with Event Bridge Configuration

Event Bridge Configuration

In this model,  we are going to create an Event Bridge Pipe and attach it to our SQS queue

Go to SQS - Click on Event Bridge Pipes and click on Connect SQS Queue to Pipe

In Event Bridge Pipe - there are four stages and each case has self-explanatory names.

  • Filtering - This lets you configure the Filter Policy to filter out selective messages from the incoming
  • Enrichment -This lets you configure any data transformation and modification and also pull data from external services like Lambda.
  • Target - Destination to send the payload to. It Can be an AWS Service, Event Bridge or any Public URL ( API Destination)

 

Target Configuration - Authentication and Transformation

One of the reasons why we cannot directly attach API Gateway with SNS is that - API Gateways do expect authentication tokens or headers

The Same Problem would arise when we are trying to attach the API Gateway as a Target to the Event Bridge Pipe.

To solve this - We need to pass the required headers or parameters to API Gateway from Event Bridge so that it does not fail to authenticate and throw a 401 error.

So Configuring the Target in the right way is necessary for the whole Event Bridge setup to work.

When you are adding API Gateway as a Target - You can define all the parameters that you want to pass to the API Gateway.

Event Bridge headers

In our case, our API Gateway (Service Proxy) and its Authorizer  looks for two header parameters for Authentication

  • X-Auth-Key
  • X-Api-Key

You can either choose to hardcode them or take them from the incoming payload using Transformation and variables.

Also, Our services look for Certain Payload Transformations done before the events reach them -  One typical transformation is to select only the message body and leave other metadata of SQS and SNS from the payload.

Here is the message sample that comes from SQS to the Event Bridge

{
  "messageId": "059f36b4-87a3-44ab-83d2-661975830a7d",
  "receiptHandle": "AQEBwJnKyrHigUMZj6rYigCgxlaS3SLy0a...",
  "body": "{\"event_type\":\"FlowControl\",\"source_service\":\"WebApp\",\"destination_service\":\"service-b\",\"X-Auth-Key\":\"\TestAuthKey",\"X-Api-Key\":\"\TestAPIKey",\"X-Api-Version\":\"v1\",\"jobname\":\"job1\"}",
  "attributes": {
    "ApproximateReceiveCount": "1",
    "SentTimestamp": "1545082649183",
    "SenderId": "AIDAIENQZJOLO23YVJ4VO",
    "ApproximateFirstReceiveTimestamp": "1545082649185"
  },
  "messageAttributes": {},
  "md5OfBody": "e4e68fb7bd0e697a0ae8f1bb342846b3",
  "eventSource": "aws:sqs",
  "eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:my-queue",
  "awsRegion": "us-east-2"
}

But all we need is the Message.Body nothing else.

So during the transformation - we can remove the other metadata information and just send this

{
  "event_type": "FlowControl",
  "source_service": "WebApp",
  "destination_service": "service-b",
  "X-Auth-Key": "SomeAuthKey",
  "X-Api-Key": "SomeAPIKey",
  "X-Api-Version": "v1",
  "jobname": "job1"
}

and you can refer to every property in this JSON like X-Auth-Key and X-Api-Key  across the parameters.

Note*: We have taken one sample service named FlowControl which expects the jobname as a path parameter

For our API Gateway and a selected service - we need four parameters

  • X-Auth-Key - for Authentication in Headers
  • X-Api-Key - for Authentication in Headers
  • {version} - for Route Selection in Path parameter
  • {jobname} - for Routing and forwarding to underlying Service in Path parameter

As you can see in the following Diagram - we are taking all of these values from the Target Input Transformer itself using Dynamic Variables like

  • $.body.X-Api-Version
  • $.body.jobname
  • $body.X-Auth-Key
  • $body.X-Api-Key

Here is the Complete Event Bridge Configuration page screenshot that covers everything we have done.

Setup Validation and Testing

Once everything is set up correctly.

When you push the message to SNS with the right payload - It would be consumed by the SQS and then picked by the Event Bridge, Transformed using our rules and pushed to the API gateway.

Once the API Gateway receives the request with the right Authorization Headers and Version and JobName information - After validation by the Authorizer. The call shall be forwarded to the internal EKS service behind the NLB

If you got the request to the Service - Kudos we got everything right.

 ProTip*:  While creating the SNS payload for this setup - try to click on the raw message delivery option so that the payload size is small and SNS metadata is not sent to SQS.

Performance Comparision

EventBridge + API Gateway

  • Concurrency: Both EventBridge and API Gateway can handle a high number of concurrent requests. API Gateway scales automatically to handle traffic, providing high concurrency.
  • Latency: API Gateway has low latency, but there is a small overhead added by EventBridge routing.
  • Cold Start: No cold start issues with EventBridge and API Gateway as they do not rely on serverless computing.

SNS + Lambda

  • Concurrency: Lambda has a default concurrency limit (which can be increased) but may be subject to throttling if the limit is exceeded.
  • Latency: Lambda functions can experience cold starts, which can add latency, especially if the function is not frequently invoked.
  • Resource Consumption: Lambda costs are based on the amount of memory allocated and the duration of execution, making it variable.

Cost Comparision

  • EventBridge + API Gateway:
    • Cost: Approximately $0.055 for 10,000 requests
    • High concurrency and low latency, no cold start issues
    • Fixed cost per request
  • SNS + Lambda:
    • Cost: Approximately $0.0095 for 10,000 requests
    • Possible cold start latency and concurrency limits
    • Dynamic cost based on payload size, execution time, and cold starts

Cost and Performance Verdict

  • Performance: EventBridge + API Gateway offers more consistent performance and higher concurrency without cold start issues.
  • Cost: SNS + Lambda is more cost-effective for 10,000 requests, but costs can vary based on execution time and memory usage.

If cost is a critical factor and the variability is acceptable, SNS + Lambda might be preferable. For higher and more consistent performance, EventBridge + API Gateway is a better choice.

 

Conclusion

It took me a long time to write about it than to create it and collectively I agree its a moderately complex setup, but with one step at a time - I am sure You can do this.

If you have any questions or have any feedback or alternate approach. Please do let me know in the comments or ping me on LinkedIn

Follow me on Linkedin My Profile
Follow DevopsJunction onFacebook orTwitter
For more practical videos and tutorials. Subscribe to our channel

Buy Me a Coffee at ko-fi.com

Signup for Exclusive "Subscriber-only" Content

More from Middleware Inventory

  • SNS TO S3
    How to Send SNS messages to S3 - using AWS Kinesis Data Firehose

    In this article, we are going to see how to save the SNS messages to S3 using Kinesis Data Firehose The first step of building a data engineering pipeline in your organization is storing the unorganized data with long retention and partitioning. Therefore storing the SNS data with Long-term storage…

  • AWS change instance type - From Console or AWS CLI

    How to Change AWS Instance Type or EC2 instance type is more interesting and frequently asked question by many AWS Professionals. We create EC2 instances as t2, t3.micro but soon we realize that our requirements just got bigger for good and we need more CPU and RAM on our instance.…

  • AWS CLI EFS List
    AWS CLI List EFS Filesystem sort by Size

    In this post we are going to see an aws cli command to list elastic file systems sorted by Size in ascending or descending order. While am sharing all the new tricks in AWS CLI I come across. this is something I have learnt recently There is a dedicated post…

  • Find Who owns the Private IP in AWS | DevOps Junction

    At times of troubleshooting an issue or doing a forensic on a security incident in your AWS account.  Finding who owns Private IP plays a major role. Do you know Almost all the AWS resources like Load Balancers, RDS, EFS has their own private IP address. So how to find…

  • AWS CLI EC2 Stop and Terminate command Examples

    In AWS infrastructure, We create a lot of EC2 instances on demand and we tend to forget about the stopped instances considering that it is stopped and not being billed for. But the fact is that Even the stopped instances would cost you a couple of pennies in various forms…