Improve security and simplify remote server access with AWS Session Manager

Remote server access is an important attack vector for cyber criminals to penetrate a system. This type of access is typically for admin purposes and with escalated privilege. Yet, research indicates that the vast majority of organisations mismanage credentials for remote access to servers over SSH.

According to the research “61% .. don’t limit or monitor the number of administrators who manage SSH, while only 35% enforce policies that prohibit SSH users from configuring their authorized keys”

In this article we will compare SSH and AWS Session Manager and describe how to implement AWS Session Manager to secure and simplify remote server access. Finally we list security risks and how to secure AWS Session Manager implementations.

Using SSH

For many years accessing EC2 instances has been through SSH. The great thing about SSH is how ubiquitous it is. With Linux and Mac OS offering command line ssh clients, Windows embracing Linux with WSL, and SSH apps for mobile devices you can ssh to an instance from anywhere.

The not so good things about this approach:

1. you open an incoming port (by default 22) which becomes available to attackers and you have to harden it. Typically, we address this with the use of an SSH Bastion instance, a dedicated and hardened instance acting as an intermediary and offering indirect SSH connectivity to servers by using SSH forwarding.
2. SSH Bastions need hardening and management and could become a single point of failure, if miss-configured.
3. Regardless of network protection, SSH typically relies on ssh key pairs to protect from brute attacks on authentication credentials. However, rotating and managing this keys as the research I referenced earlier shows is a key risk. This in line with my own experiences. A month into a new job as a startup CTO, I found out that leavers could still access systems because their SSH public keys had not been removed and their home IP addresses were still in EC2 security groups.
4. On AWS, you can’t rely on Cloud trail to audit or respond to unauthorized intrusions since CloudTrail only monitors AWS API calls.

Using SSH, even in its Bastion form fails to meet two principles of the Security Pillar of the AWS Well-Architected Framework:

• Enable Traceability
• Implement a strong identity foundation

Using SSM Session Manager

This is where AWS SSM Session Manager offers a great alternative that is both secure and more usable. Session Manager securely tunnels your terminal connections over HTTPS/TLS and allows you to quickly start terminal sessions from the browser or the SSM Session Manager plugin. Like SSH the Session Manager supports port forwarding. The connections are authenticated with AWS credentials and are via the SSM AWS API. As a result:

1. No need to open any incoming ports on your instance
2. No need for Bastion hosts or any other resources to manage.
3. Connections are logged in CloudTrail, offering auditability, but also the ability to respond to intrusions via EventBridge events and your own rules.
4. Sessions can be logged in S3 buckets or CloudWatch logs for troubleshooting purposes.
5. No need for SSH keys; it uses standard AWS authentication and is subject to IAM policies.

When enabled, Session logging is down to keystroke level . You should make sure adequate data protection is in place to avoid sensitive data leakages. See below (“Securing Session Manager Implementations”) on how to address this.

Setting up SSM Session Manager is simple. It requires:

1. IAM Permissions for the EC2 role.
2. SSM agent installed on your instance (in most cases already per-installed)

3. Outbound network connectivity to SSM related endpoints.

IAM Permissions

The role you will use in your EC2 profile should have sufficient SSM permissions to use the Session Manager. The documentation suggests that AWS-managed policy AmazonSSMManagedInstanceCore offers these permissions but at the time of writing the policy has no Session Manager permissions. Instead, the following policy can be used:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowSessionManagerSupportingPrequisites",
            "Effect": "Allow",
            "Action": [
                "ssm:GetConnectionStatus",
                "ssm:DescribeSessions",
                "ssm:DescribeInstanceInformation",
                "ssm:UpdateInstanceInformation",
                "ssm:DescribeInstanceProperties",
                "ssmmessages:CreateControlChannel",
                "ssmmessages:CreateDataChannel",
                "ssmmessages:OpenControlChannel",
                "ssmmessages:OpenDataChannel",
                "ec2messages:SendReply",
                "ec2messages:GetMessages",
                "ec2messages:GetEndpoint",
                "ec2messages:FailMessage",
                "ec2messages:DeleteMessage",
                "ec2messages:AcknowledgeMessage",
                "ec2:DescribeInstances"
            ],
            "Resource": "*"
        },
        {
            "Sid": "AllowSSMSessionManagerSessions",
            "Effect": "Allow",
            "Action": "ssm:StartSession",
            "Resource": "arn:aws:ec2:*:*:instance/*"
        },
        {
            "Sid": "AllowCurrentUserToResumeOrTerminateASession",
            "Effect": "Allow",
            "Action": [
                "ssm:ResumeSession",
                "ssm:TerminateSession"
            ],
            "Resource": "arn:aws:ssm:*:*:session/${aws:username}-*"
        }
    ]
}

To enable port forwarding you will need to add the following statement

{
  "Sid": "SessionManagerPortForward",
   "Effect": "Allow",
   "Action": "ssm:StartSession",
   "Resource":
     [ 
       "arn:aws:ssm:*::document/AWS-StartPortForwardingSession"
     ]
}

Session Manager also supprort SSH over its sessions. This requires to add an SSM document as a resource of the StartSession permission, e.g.

{
"Sid": "SSHOverSessionManager",            
"Effect": "Allow",
"Action": "ssm:StartSession",
            "Resource": [
                "arn:aws:ssm:*:*:document/AWS-StartSSHSession"
            ]
        }
    ]

The last statements above do not have a “arn:aws:ec2:*:*:instance/*”, in their resoure clause, as they assume this was done in the main AllowSSMSessionManagerSessions sid.

Instance Setup

Most people, switch off as soon as they read SSM agent installation. This is a shame because the agent is already pre-installed in most common AMIs. AWS do not do themselves a favour by not having a quick start up in addition to the detailed documentation. The AMIs with the SSM agent pre-installed are:

• Amazon Linux
• Amazon Linux 2 (but not necessarily custom AMIs based on Amazon Linux that may have removed it, so always double check Amazon Linux2 custom AMIs)
• Amazon Linux 2 ECS-Optimized Base AMIs
• SUSE Linux Enterprise Server (SLES) 12 and 15
• Ubuntu Server 16.04, 18.04, and 20.04

If your AMI does not come with the SSM Agent then you do need to install it, which is relatively straightforward and consists of downloading the installation artifact (deb, rpm, etc), installing it, and enabling the service. Full instructions for most OSes can be found in the AWS Documentation

Setting up Networking Connectivity for Session Manager

For instances with public internet connectivity, the security group for the instance will need to have outbound communication on port 443 to

• ec2messages.region.amazonaws.com
• ssm.region.amazonaws.com
• ssmmessages.region.amazonaws.com

In this article we will focus on the use case of a VPC with no internet connectivity.

In this case, you will need to create VPC Endpoints for the above endpoints . All three support VPCE Interfaces and you can easily create interfaces in your VPC (remember to reverse the host name, e.g. com.amazonaws.<region>.ssm or just search the service name).

Here is an example creating the ssm endpoint which i named ssm-endpoint, on the AWS console from VPC, Endpoints, Crate Endpoint and after selecting AWS Services:

Select the VPC and subnet(s)

then apply any network restrictions in a security group or VPCE policy

It is important that the EC2 instance is launched on the selected subnets and the security group should not block access from the instance. If you have a restricted VCPE Policy in the additional settings, the EC2 Instance Profile role should be allowed to have access.Once the VPCE interfaces are configured you need to ensure outbound connections to the VPCE are allowed by your security group of the instance.

Note that these three endpoints are VPCE interfaces and unlike VPCE Gateways (e.g. S3 or DynamoDb) you do not need to whitelist VPCE specific addresses; instead you need to whitelist the subnets the VPCE interfaces are attached to; if for some (usually bad) reason one VPCE interface is attached to one subnet and the others to another one, you need to whitelist a CIDR that covers both.

Finally, you will need to make sure you traffic to these endpoints (and corresponding incoming ephemeral ports) is not blocked by any NACL rules.

That’s it! Launch the instance and once it’s running click on connect, and session manager (some times the first few minutes after launch the tab may have an error, but it eventually becomes available)

Click connect again and you should have an interactive terminal in your browser

By default, you are logged on as the ssm-user but you can switch to other users. e.g.

sudo su - ec2-user

or even get root access

sudo su -

You can also access the your instance via the Session Manager session of the SSM service in the AWS Console or the SSM plugin. The plugin gives you SSH-like terminal access and it is very useful when port forwarding is needed. It’s an excellent way for instance to pen test web apps located deep in private networks without opening any ports at your virtual network perimeter.. For more info see https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html

Routing SSH and SCP over SSM

Once the SSM Plugin is installed you can even proxy SSH and SCP over SessionManager, assuming 1) ssh is running n your instance and 2) you have the StartSession permission on the SSH SSM document as described earlier .

This is achieved by adding a proxy command in the ssh config file (typically ~/.ssh/config) to proxy SSH/SCP commads to instances over SSM. eg.

# SSH over Session Manager
host i-* mi-*
 ProxyCommand sh -c “aws ssm start-session — target %h — document-name AWS-StartSSHSession — parameters ‘portNumber=%p’”

For more detail see https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-getting-started-enable-ssh-connections.html

Securing Session Manager implementations

Although, Session Manager is a security improvement over SSH, there are steps you should take to ensure a secure implementation:

1. Logging Sessions could leak private data. Update your guidance to your staff to avoid use of Session Managers to display sensitive data. Save any session logs (but also CloudTrail trails) SSE Encrypted with your own Customer-Managed Key and secure Session logs at the same level as you would for sensitive data. Look at the use of the Amazon Comprehend API to search logs for Personal Data. The later is NOT a substitute for defensive security and could entail writing scripts similar to this AWS demo https://aws.amazon.com/blogs/machine-learning/detecting-and-redacting-pii-using-amazon-comprehend/
2. The ssm-user used by Session Manager for its interactive sessions is by default a privileged user. You can (and should) restrict what that user can do by restricting the commands available in a session and/or switching off its admin permissions depending on your requirements.
3. Apply Lest Privilege to the IAM Role . The sample policy above gives the role access to create sessions on any instance. You should restrict that to the instances required. Use condition with tags to restrict it to a group of instances rather than individual instance if you want to reuse it rather than copy and paste. You can also further restrict — based on your use cases with conditions such region, caller IP, etc.
4. Remove port forwarding permissions from the IAM policy if port forwarding is not needed.
5. Apply the same principle at the network layer. In addition to the EC2 Instance Security Group, secure the VPCE with its own security group restricting incoming traffic and use the VPCE Policy to whitelist only roles that need access to these points. A good approach would be to have a security group for the VPCE with the allowed subnets as incoming and then use this Security Group in the outgoing rule of the EC2 instance Security Group. This way, you avoid duplicating values in two different places and both interfaces are protected.
6. Take advantage of the Cloud Trail integration and create alarms and events to watch for intrusions.
7. Automate all of these steps to avoid mis-configurations that could result to denial of service or accidental unauthorized access.