In this article, we will see an interesting use case or system design where we want to connect AWS SNS to an internal HTTP or HTTPS service.
SNS is a public service designed to connect to external public HTTPS endpoints.
SNS is purpose-built for external communications and it supports Public HTTP/HTTPS endpoints to subscribe directly
What if you want to add an internal HTTP endpoint from your VPC as a subscription - you need to workaround with Proxies
There are two types of Proxies we can use.
- Lambda Proxy - Using Lambda Function as a Proxy
- Event Bridge and API Gateway Service Proxy - Using Event Bridge Pipe and API Gateway as a Proxy
Lambda Proxy
In this model, a Lambda function acts as a Proxy to forward the calls to internal services and HTTP endpoints
In the following diagram, you can see that Lambda is added as a subscription to the SNS
Here, Lambda does three things
Public to Private Conversion
The first and Primary responsibility of this pattern - is to convert the Public connection to a private connection and thereby able to connect to all the internal Endpoints and URLs - This Service Proxy Lambda would be inside the VPC and run on the private Subnets and can DNS resolve internal Route 53 entries
Enrichment ( Validation & Transformation)
In this stage - Lambda can optionally parse the SNS payload. Validate and transform it to another schema accepted by the destination service.
Adding additional headers or inputs based on the Source or event_type
attributes, which might be needed by the destination service for authentication or core functionality
Logging and Observability can also be enabled along with data sanitization if need be
But this is optional, If we want to forward the SNS message to the HTTP service as is, this can be skipped.
Authentication
This may be optional too - but in a few cases, we might want to add some Authorization token or header before passing the request to the internal service.
We might want to take some info from the Passed payload from SNS and forward them as headers like X-API-Key
or X-Origin-Service
for authentication
Here is a sample Lambda Proxy that does all three things.
import json import os import re import requests def lambda_handler(event, context): try: # Read SNS payload sns_message = event['Records'][0]['Sns']['Message'] payload = json.loads(sns_message) # Enrich: Validate JSON schema if not validate_payload(payload): raise ValueError("Invalid payload schema") # Authenticate: Extract headers and remove them from payload headers = { 'X-User-Email': payload.pop('X-User-Email', None), 'X-User-Token': payload.pop('X-User-Token', None) } if None in headers.values(): raise ValueError("Authentication headers missing in payload") # Route: Determine the target service based on event_type event_type = payload.get('event_type') target_url = get_target_url(event_type) if not target_url: raise ValueError("Invalid event_type or target service not defined") # Forward the request to the target service response = requests.post(target_url, headers=headers, json=payload) return { 'statusCode': response.status_code, 'body': response.text } except Exception as e: return { 'statusCode': 500, 'body': str(e) } def validate_payload(payload): """ Validates the payload schema. Modify this function based on your schema requirements. """ required_fields = {'event_type', 'X-User-Email', 'X-User-Token'} if not required_fields.issubset(payload.keys()): return False if not re.match(r"[^@]+@[^@]+\.[^@]+", payload['X-User-Email']): return False return True def get_target_url(event_type): """ Determines the target service URL based on event_type. """ service_urls = { 'serviceA': os.getenv('serviceA'), 'serviceB': os.getenv('serviceB'), 'serviceC': os.getenv('serviceC') } return service_urls.get(event_type) # Sample event structure for testing locally if __name__ == "__main__": sample_event = { "Records": [ { "Sns": { "Message": json.dumps({ "event_type": "serviceA", "X-User-Email": "[email protected]", "X-User-Token": "token123", "data": "sample data" }) } } ] } print(lambda_handler(sample_event, None))
This is one of the patterns where Lambda acts as a Proxy and with Lambda being serverless - this architecture can scale to meet growing demands in future and handle higher concurrency too.
But Lambda has a few concerns like Cold start, Concurrency limit and Dynamic Pricing based on resource consumption etc.
Let us explore the other solution now.
Event Bridge & API Gateway Service Proxy
In this alternate approach. Instead of Lambda, we are going to use API Gateway as a Proxy.
While you can use the Public API Gateway as an HTTPS subscription directly with SNS - One Problem might arise with Authentication.
SNS does not support Authenticated HTTP/HTTPS endpoints - There is no provisioning to define the Authentication Headers or Keys during the creation of Subscription
So we need some additional wrapper service that enables us to Authenticate with the API Gateway
That's where we bring the Event Bridge Pipe and SQS.
SNS cannot directly communicate with the API Gateway or Event Bridge either - ( There is firehose but that is a different implementation) so we need SQS here as a Connector
Refer to the following System Design architecture with SQS and EventBridge
Steps Involved
To achieve this system design the following steps are involved.
- Create an SQS queue and use it for the SNS subscription
- In the SQS, Create an Event Bridge Pipe
- In Event Bridge we can do four things
- Filter - In case we want to filter out certain messages
- Enrich - For Payload Transformation - Essential for our current requirement as we need to read the payload get the data and use it for Authentication
- Authenticate - You can define what are the headers to be sent to the API Gateway
- Target - Define API gateway as the Target ( as of now only public API gateways are supported in Event Bridge)
- Create the API Gateway with a Swagger that acts as a Service Proxy
- Create VPC links and connect to NLBs inside our VPC - In our case, these NLBs are directly bound with the EKS service
We will not cover all the steps, but I will provide the details and configuration elements required to implement this setup.
The Core elements of this design are Event Bridge Pipe and API Gateway and how to configure the Parsing and Authentication lets explore it a little deeper.
Creating an SNS Subscription, creating a VPC Link inside the API gateway, and creating NLBs via Kubernetes all require some prerequisite knowledge of the respective areas.
You might be already aware - If not, Please refer to the following links
- Creating SNS Subscription
- Creating NLB and VPC Link for the API gateway
- Creating Network Load Balancer with Service in EKS
I hope you have the SNS, SQS and the NLB along with VPC links required before proceeding further.
Now let us switch to our Objective on learning how to use the internal NLB service as a Subscription with SNS
Before we configure the Event Bridge - we need an API gateway as a prerequisite so let's cover that first
API Gateway Configuration
Our API Gateway Setup is going to consist of the following parts
- Authorizer - A Lambda function, written in Python to Authorize the request based on the presence of specific headers or query string - in our case its headers
- Swagger File - Open API Specification Swagger File for Creating the REST API contains the API endpoints and methods and Authorizer reference.
Following is the Technical diagram of how our API Gateway looks and functions.
You can download the Swagger file and the Python Authorizer Lambda function code from the following Github Repository link
As you clone this repository you can find the Authorizer Source Code and API Gateway Swagger file in their respective directories.
In VS Code - you can use the Swagger Viewer Plugin to view the Swagger file in a nice API documentation format
This is how it looks on the VS Code with Swagger Viewer
You must replace the Endpoints and Methods, Parameters and The Authorizer ARN inside the Swagger file before using it. This is just a sample swagger for reference
Once you have created the Service Proxy API gateway and the Authorizer Lambda and configured the VPC links and NLBs. you are good to go forward with Event Bridge Configuration
Event Bridge Configuration
In this model, we are going to create an Event Bridge Pipe and attach it to our SQS queue
Go to SQS - Click on Event Bridge Pipes and click on Connect SQS Queue to Pipe
In Event Bridge Pipe - there are four stages and each case has self-explanatory names.
- Filtering - This lets you configure the Filter Policy to filter out selective messages from the incoming
- Enrichment -This lets you configure any data transformation and modification and also pull data from external services like Lambda.
- Target - Destination to send the payload to. It Can be an AWS Service, Event Bridge or any Public URL ( API Destination)
Target Configuration - Authentication and Transformation
One of the reasons why we cannot directly attach API Gateway with SNS is that - API Gateways do expect authentication tokens or headers
The Same Problem would arise when we are trying to attach the API Gateway as a Target to the Event Bridge Pipe.
To solve this - We need to pass the required headers or parameters to API Gateway from Event Bridge so that it does not fail to authenticate and throw a 401 error.
So Configuring the Target in the right way is necessary for the whole Event Bridge setup to work.
When you are adding API Gateway as a Target - You can define all the parameters that you want to pass to the API Gateway.
In our case, our API Gateway (Service Proxy) and its Authorizer looks for two header parameters for Authentication
- X-Auth-Key
- X-Api-Key
You can either choose to hardcode them or take them from the incoming payload using Transformation and variables.
Also, Our services look for Certain Payload Transformations done before the events reach them - One typical transformation is to select only the message body and leave other metadata of SQS and SNS from the payload.
Here is the message sample that comes from SQS to the Event Bridge
{ "messageId": "059f36b4-87a3-44ab-83d2-661975830a7d", "receiptHandle": "AQEBwJnKyrHigUMZj6rYigCgxlaS3SLy0a...", "body": "{\"event_type\":\"FlowControl\",\"source_service\":\"WebApp\",\"destination_service\":\"service-b\",\"X-Auth-Key\":\"\TestAuthKey",\"X-Api-Key\":\"\TestAPIKey",\"X-Api-Version\":\"v1\",\"jobname\":\"job1\"}", "attributes": { "ApproximateReceiveCount": "1", "SentTimestamp": "1545082649183", "SenderId": "AIDAIENQZJOLO23YVJ4VO", "ApproximateFirstReceiveTimestamp": "1545082649185" }, "messageAttributes": {}, "md5OfBody": "e4e68fb7bd0e697a0ae8f1bb342846b3", "eventSource": "aws:sqs", "eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:my-queue", "awsRegion": "us-east-2" }
But all we need is the Message.Body
nothing else.
So during the transformation - we can remove the other metadata information and just send this
{ "event_type": "FlowControl", "source_service": "WebApp", "destination_service": "service-b", "X-Auth-Key": "SomeAuthKey", "X-Api-Key": "SomeAPIKey", "X-Api-Version": "v1", "jobname": "job1" }
and you can refer to every property in this JSON like X-Auth-Key
and X-Api-Key
across the parameters.
Note*: We have taken one sample service named FlowControl which expects the jobname
as a path parameter
For our API Gateway and a selected service - we need four parameters
- X-Auth-Key - for Authentication in
Headers
- X-Api-Key - for Authentication in
Headers
- {version} - for Route Selection in
Path
parameter - {jobname} - for Routing and forwarding to underlying Service in
Path
parameter
As you can see in the following Diagram - we are taking all of these values from the Target Input Transformer itself using Dynamic Variables like
- $.body.X-Api-Version
- $.body.jobname
- $body.X-Auth-Key
- $body.X-Api-Key
Here is the Complete Event Bridge Configuration page screenshot that covers everything we have done.
Setup Validation and Testing
Once everything is set up correctly.
When you push the message to SNS with the right payload - It would be consumed by the SQS and then picked by the Event Bridge, Transformed using our rules and pushed to the API gateway.
Once the API Gateway receives the request with the right Authorization Headers and Version and JobName information - After validation by the Authorizer. The call shall be forwarded to the internal EKS service behind the NLB
If you got the request to the Service - Kudos we got everything right.
ProTip*: While creating the SNS payload for this setup - try to click on the raw message delivery option so that the payload size is small and SNS metadata is not sent to SQS.
Performance Comparision
EventBridge + API Gateway
- Concurrency: Both EventBridge and API Gateway can handle a high number of concurrent requests. API Gateway scales automatically to handle traffic, providing high concurrency.
- Latency: API Gateway has low latency, but there is a small overhead added by EventBridge routing.
- Cold Start: No cold start issues with EventBridge and API Gateway as they do not rely on serverless computing.
SNS + Lambda
- Concurrency: Lambda has a default concurrency limit (which can be increased) but may be subject to throttling if the limit is exceeded.
- Latency: Lambda functions can experience cold starts, which can add latency, especially if the function is not frequently invoked.
- Resource Consumption: Lambda costs are based on the amount of memory allocated and the duration of execution, making it variable.
Cost Comparision
- EventBridge + API Gateway:
- Cost: Approximately $0.055 for 10,000 requests
- High concurrency and low latency, no cold start issues
- Fixed cost per request
- SNS + Lambda:
- Cost: Approximately $0.0095 for 10,000 requests
- Possible cold start latency and concurrency limits
- Dynamic cost based on payload size, execution time, and cold starts
Cost and Performance Verdict
- Performance: EventBridge + API Gateway offers more consistent performance and higher concurrency without cold start issues.
- Cost: SNS + Lambda is more cost-effective for 10,000 requests, but costs can vary based on execution time and memory usage.
If cost is a critical factor and the variability is acceptable, SNS + Lambda might be preferable. For higher and more consistent performance, EventBridge + API Gateway is a better choice.
Conclusion
It took me a long time to write about it than to create it and collectively I agree its a moderately complex setup, but with one step at a time - I am sure You can do this.
If you have any questions or have any feedback or alternate approach. Please do let me know in the comments or ping me on LinkedIn
![](https://www.middlewareinventory.com/wp-content/uploads/2023/01/SaravAK-300x300.jpeg)
Follow me on Linkedin My Profile Follow DevopsJunction onFacebook orTwitter For more practical videos and tutorials. Subscribe to our channel
![Buy Me a Coffee at ko-fi.com](https://az743702.vo.msecnd.net/cdn/kofi3.png?v=2)
Signup for Exclusive "Subscriber-only" Content