EC2 Instance Connect Endpoint - SSH the easy way | Devops Junction

But there is a new option released recently (June 2023) from the labs of AWS.  EC2 instance Connect endpoint

Before we move further into the EC2 instance connect endpoint it is vital to know all other available options and how they work

How to SSH to EC2 Instance in Private Subnet

These are the best approaches or methods to enable SSH to an EC2 instance running in Private Subnet securely.

Bastion Host

This is the most common way of connecting to an EC2 instance in a Private Subnet. You create a Bastion Host in Public Subnet then SSH to the Bastion Host and then SSH to the EC2 instance in Private Subnet. This is a very common pattern and is used in most of the organizations. The following diagram shows the architecture of this pattern

EC2 Instance Connect

EC2 Instance Connect is another better way to SSH to an EC2 instance in a Private Subnet. This provides a mechanism that uses IAM credentials to push ephemeral SSH keys to an instance, making long-lived keys unnecessary

To use EC2 Instance Connect to connect to an instance, the instance must have EC2 Instance Connect installed. You can either launch the instance using an AMI that comes preinstalled with EC2 Instance Connect, or you can install EC2 Instance Connect on instances that are launched with supported AMIs.

EC2 instance connect feature still needs a Public IP address to allow connections from the internet.

To know more about EC2 instance connect refer to this documentation

The following diagram shows the architecture of this pattern

Session Manager (SSM) Agent-based SSH connectivity

In this approach, we install an SSM agent on the EC2 instance in Private Subnet and then use SSM Session Manager to connect to the EC2 instance. This is a very secure way of connecting to the EC2 instance as it does not need any Public IP address and the connection is established via AWS Systems Manager service.

Session Manager is a fully managed AWS Systems Manager capability that lets you manage your Amazon EC2 instances through an interactive one-click browser-based shell or the AWS CLI, You can use Session Manager to start a session with an instance in your account. After the session has started, you can run bash commands as you would through any other connection type.

The following diagram shows the architecture of this pattern

Based on the understanding we had of the preceding three options, you would know that they require a lot of setup and/or a public IP address along with Agent/Plugin installation for SSM or EC2 instance connect

Now I want to bring my favourite configuration management tool Ansible here, one of the key factors for the success of Ansible over its predecessors Chef and Puppet  is indeed the agentless nature of it.

So we need some agentless connectivity here as well. That is where the EC2 instance connect endpoint comes in.

So there comes the latest approach number 4. EC2 instance Connection Endpoint

What is the EC2 Instance Connect Endpoint?

EC2 instance Connect Endpoint is a secure tunnel that supports SSH and RDP to your EC2 servers inside the VPC without the requirement of a Public IPV4 address or bastion host. ( It does not support ipv6-only servers)

EIC( EC2 Instance Connect) Endpoint is an identity-aware TCP proxy, that connects your workstation directly to the server through a WebSocket tunnel using AWS CLI client.  Once the tunnel is established you would be able to connect to services on EC2 using your local loopback IP (127.0.0.1)

With EIC Endpoint you can directly use AWS console to connect(SSH) to your servers inside VPC

Either way Authentication and Authorization are done before the traffic reaches the VPC. so it is secure.

It means you can SSH or RDP to your instance with just AWS CLI or console without having to install any agent or copy the public keys prior

Is EC2 Instance Connect Endpoint Secure?

Since the EC2 instance connect endpoint lets you connect directly from public IP space from anywhere in the world there are few fine-grained security controls/checks in place.

• To be able to create an EC2 instance Connect Endpoint - you need the following IAM permissions which are mostly given to Cloud Admins. ( Control Plane)
  1. ec2:CreateInstanceConnectEndpoint
  2. ec2:CreateNetworkInterface
  3. ec2:CreateTags
  4. iam:CreateServiceLinkedRole
• To use an EC2 instance to connect the endpoint, you would need the following permissions - There by separate access permissions for Administrator ( Control Plane) and user ( Data Plane) is implemented
  1. ec2-instance-connect:OpenTunnel
  2. ec2-instance-connect:SendSSHPublicKey
  3. ec2-instance-connect:SendSerialConsoleSSHPublicKey
• Besides this, there are a lot more restrictions that you can apply to the IAM policy such as
  • Restricting access to certain source IPs
  • Allowing access to specific instances based on IP or Tag
  • Restrictions based on the Destionation IP and/or Port for the server - you can choose to allow only selected ports like 22 over the EIC endpoint
• Audit Trail logs for both successful and failed attempts are captured by default in Cloud Trail - Here is the sample event data you can find for every single EIC endpoint action

{
    "eventVersion": "1.08",
    "userIdentity": {
        "type": "IAMUser",
        "principalId": "**************",
        "arn": "arn:aws:iam::**************:user/sarav",
        "accountId": "**************",
        "accessKeyId": "**************",
        "userName": "sarav"
    },
    "eventTime": "2023-07-25T09:00:37Z",
    "eventSource": "ec2-instance-connect.amazonaws.com",
    "eventName": "SendSSHPublicKey",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "**************",
    "userAgent": "aws-cli/2.13.3 Python/3.11.4 Darwin/22.1.0 exe/x86_64 prompt/off command/ec2-instance-connect.ssh",
    "errorCode": "AccessDenied",
    "errorMessage": "User: arn:aws:iam::**************:user/sarav is not authorized to perform: ec2-instance-connect:SendSSHPublicKey on resource: arn:aws:ec2:us-east-1:**************:instance/i-************** because no identity-based policy allows the ec2-instance-connect:SendSSHPublicKey action",
    "requestParameters": null,
    "responseElements": null,
    "requestID": "**************",
    "eventID": "**************",
    "readOnly": false,
    "eventType": "AwsApiCall",
    "managementEvent": true,
    "recipientAccountId": "**************",
    "eventCategory": "Management",
    "tlsDetails": {
        "tlsVersion": "TLSv1.3",
        "cipherSuite": "TLS_AES_128_GCM_SHA256",
        "clientProvidedHostHeader": "ec2-instance-connect.us-east-1.amazonaws.com"
    }
}

How to Create EC2 Instance Connect Endpoint

To create the EC2 instance connect endpoint, you need to have a set of IAM permissions as we have discussed earlier.

Once the IAM permissions are ready you can create an EIC endpoint in two ways

1. AWS Management Console
2. AWS CLI ( version 3.11 and above)

Creating EC2 Instance connect endpoint using AWS Console

aws ec2 create-instance-connect-endpoint 
--region <desired-region>
--subnet-id <your-subnet> 
--security-group-id <security-group-with-specific-instances-and-ports>

When you create an EC2 Instance Connect Endpoint, a service-linked role is automatically created for the Amazon EC2 service in AWS Identity and Access Management (IAM).

Amazon EC2 uses the service-linked role to provision network interfaces in your account, which is required when creating EC2 Instance Connect Endpoints.

that's it, you have now successfully created the EC2 instance connect endpoint

Using EC2 Instance Connect Endpoint ( EICE)

Now we have created the EIC Endpoint, now we can go ahead and use it practically. With the inception of EC2 instance connect endpoint

There are two new sub-commands added to the aws ec2-instance-connect  CLI command

• ssh - to open an interactive SSH session to the remote EC2 instance
• open-tunnel - to open a tunnel on port

ssh - to open an interactive SSH session

This ssh subcommand is newly added to the ec2-instance-connect command suite to exclusively support the EC2 instance connect endpoint

With this, you would be able to open an interactive SSH session to the remote server on a private subnet using EICE

here is the syntax of the command

aws ec2-instance-connect ssh 
--instance-id i-*********
--os-user ubuntu

With the preceding command, you would be able to open an SSH session with the remote server on port 22 through EICE

Here is the screenshot of EICE based ssh session, when I tried at my end

There are more options to this command which you can refer to and try out from the documentation

What if you want to connect to an EC2 server which uses a different port for SSH other than 22

As of now ( July 2023) only port 22 and 3389 are allowed through EICI ssh and open-tunnel commands

It was actually available on all the ports from the first release date of June 24 2023 but then was revoked and restricted to only 22 and 3389 (default ports of SSH and RDP) on Jun 28 2023.

You might see the following error if you are trying to use a different port

~|⇒ aws ec2-instance-connect ssh – instance-id i-******** – ssh-port 2222
2023-07-25 20:05:29,234 - awscli.customizations.ec2instanceconnect.websocket - ERROR - {"ErrorCode":"InvalidParameter","Message":"The specified RemotePort is not valid. Specify either 22 or 3389 as the RemotePort and retry your request."}

AWS_ERROR_HTTP_WEBSOCKET_UPGRADE_FAILURE: Failed to upgrade HTTP connection to Websocket.
kex_exchange_identification: Connection closed by remote host
Connection closed by UNKNOWN port 65535

Let's hope they enable other ports soon. I will try to keep this document updated.

Meanwhile, you can refer to these forums and links for further updates

• https://repost.aws/questions/QUZgD8nmZGTR-G1NL5-zFbaw/ec2-instance-connect-endpoint-blocks-ports-other-than-22-and-3389
• https://repost.aws/questions/QU_h42-ck0R-alITadXrrXSQ/rds-configuration

open-tunnel - to open an SSH/RDP tunnel

With the open-tunnel command, we can open a secure tunnel to the remote port ( 22 or 3389) and map to the local port - Just like port forwarding

Since AWS has removed/blocked the other ports this feature can be used only for SSH or RDP connection tunneling

To test it you can first create a tunnel with the following command to a remote port 22 and choose any local port, in my case, I am using 8080

aws ec2-instance-connect open-tunnel 
--instance-id i-******** 
--remote-port 22 
--local-port 8080

once the tunnel is open you can use the following command to SSH to the remote EC2 instance

ssh ubuntu@localhost -p 8080

Yes, you just used localhost  and port 8080 to SSH to your remote EC2 instance as the tunnel is forwarding to the remote host and port

Here is a screenshot of me trying a few commands using this tunnel

For the brief period when AWS did not block the other ports there were a lot of use cases and hacks engineers were able to attain and here is one of them

• https://www.doit.com/secured-access-to-private-rds-using-amazon-ec2-instance-connect-endpoint/

I agree that allowing all ports would actually open up a window for a lot of security risks and hacks as aws states ec2-instance-connect command is designed for allowing SSH connection and it would remain this way for a while at least

Meanwhile, Stay connected and I will keep this article updated

Cheers
Sarav AK

Follow me on Linkedin My Profile
Follow DevopsJunction onFacebook orTwitter
For more practical videos and tutorials. Subscribe to our channel

Signup for Exclusive "Subscriber-only" Content

Name*

Email*