Set Up and Destroy an SSH-able EC2 Instance with Ansible

May 3, 2023

I’ve been sort of relearning how to use Ansible and trying to improve my automations. To that end, I’m building out a project from the ground up to try to fill in missing holes in my knowledge and maybe learn some new tricks.

In this article, I’ll create an Ansible playbook that:

creates an EC2 instance
creates SSH keys for that instance
configures my laptop to be able to SSH in immediately
destroys the instance and keys, and removes the corresponding config from my laptop

This will be a pretty simple playbook, and security is not a priority yet, so there will be a few security issues and drawbacks that I’ll talk about at the end. I plan to keep revising the playbook to make incremental improvements.

Step By Step Creation

To do the EC2 creation, the playbook will carry out these steps:

1. Create SSH keys on AWS and copy the private key to the local machine.

- name: Create an EC2 key
  block:
    - name: Create the EC2 key
      amazon.aws.ec2_key:
        name: "{{ ssh_key_name }}"
        region: "{{ aws_region }}"
        force: true
      register: ec2key
    - name: Save the private key locally
      ansible.builtin.copy:
        content: "{{ ec2key.key.private_key }}"
        dest: "{{ ssh_key_path_local }}"
        mode: 0600
      when: ec2key.changed
  tags: ['create']

2. Create a security group that will allow SSH access from any IP address.

AWS is “Allow Only”, so we need to whitelist our access.

- name: Create EC2 ssh security group
  block:
  - name: Create security group
    amazon.aws.ec2_security_group:
      name: "{{ host_name }} SSH"
      description: SSH access for instance
      region: "{{ aws_region }}"
      rules:
        - proto: tcp
          ports: 22
          cidr_ip: 0.0.0.0/0
          rule_desc: allow all ssh on port 22
    register: security_group
  tags: ['never', 'create']

3. Create an instance using the created SSH keys and security group.

We also configure User Data to add passwordless sudo for the “ubuntu” user. User Data runs once on the initial launch as root. It can be configured to run on every reboot.

- name: Create EC2 Block
  block:
  - name: Create EC2 instances
    amazon.aws.ec2_instance:
      name: "{{ ssh_key_name }}"
      region: "{{ aws_region }}"
      availability_zone: "{{ availability_zone }}"
      key_name: "{{ ssh_key_name }}"
      instance_type: "{{ instance_type }}"
      image_id: "{{ image_id }}"
      user_data: |
        #!/bin/bash
        sudo apt update && sudo apt upgrade
        sudo awk '/root\s+ALL=\(ALL:ALL\) ALL/ {print; print "ubuntu ALL=(ALL) NOPASSWD:ALL"; next}1' /etc/sudoers | sudo tee /etc/sudoers        
      security_groups: ['default', "{{ security_group.group_id }}"]
      network:
        assign_public_ip: true
    register: ec2

4. Wait for SSH access to come up.

Sometimes I’m getting some failures where AWS seems to be hanging or taking a little longer to do things. Occasionally, the playbook fails here because the previous step has “completed”, but the instance isn’t available yet.

I haven’t investigated this yet, but I figure it partly has to do with my frequent create/destroy cycle.

- name: Wait for ssh to come up
  local_action:
    module: wait_for
    host: "{{ item.public_ip_address }}"
    port: 22
    delay: 10
    timeout: 120
  loop: "{{ ec2.instances }}"

5. Update the local ~/.ssh/config.

This allows immediate access into the instance via SSH.

- name: Update local .ssh/config
  ansible.builtin.blockinfile:
    path: ~/.ssh/config
    backup: "{{ backup_ssh_config }}"
    block: |
      Host {{ host_name }}
          IdentityFile {{ ssh_key_path_local }}
          IdentitiesOnly yes
          User ubuntu
          HostName {{ item.public_ip_address }}
          StrictHostKeyChecking no
          UserKnownHostsFile ~/.ssh/known_hosts_amz      
  loop: "{{ ec2.instances }}"

6. Make sure there is a group line in the Ansible hosts file.

This usually looks something like:

[webservers]

Do it in Ansible like this:

- name: Ensure there is a group line for our instances in hosts file
  ansible.builtin.lineinfile:
    path: ./hosts
    line: "[{{ inventory_host_groups }}]"
    state: present

7. Add our new instance to the Ansible hosts file after the group line.

- name: Update local ansible hosts file
  ansible.builtin.lineinfile:
    path: ./hosts
    line: "{{ host_name }}"
    insertafter: "[{{ inventory_host_groups }}]"
    state: present

I’m using Ansible tags to specify tasks that should be executed on create and destroy.

Step By Step Destruction

Destruction of the server is pretty simple. We start by finding the instance and then reverse most of the changes we made on creation.

Note that I’m not removing the group line from the Ansible hosts file.

- name: Terminate EC2 Block
  block:
  - name: Get EC2 instance info
    amazon.aws.ec2_instance_info:
      filters:
        "tag:Name": "{{ host_name }}"
    register: instance
  - name: Terminate EC2 instances
    amazon.aws.ec2_instance:
      state: terminated
      name: "{{ ssh_key_name }}"
    register: terminated_instances
  - name: Remove the EC2 key
    amazon.aws.ec2_key:
      name: "{{ ssh_key_name }}"
      state: absent
  - name: Remove security group
    amazon.aws.ec2_security_group:
      name: "{{ host_name }} SSH"
      state: absent
  - name: Delete local ssh key
    ansible.builtin.file:
      path: "{{ ssh_key_path_local }}"
      state: absent
  # TODO Remove known hosts entry
  - name: Update local ansible hosts file
    ansible.builtin.lineinfile:
      path: ./hosts
      search_string: "{{ host_name }}"
      state: absent
  - name: Remove block from local .ssh/config
    ansible.builtin.blockinfile:
      path: ~/.ssh/config
      state: absent
      block: |
        Host {{ host_name }}
            IdentityFile {{ ssh_key_path_local }}
            IdentitiesOnly yes
            User ubuntu
            HostName {{ item.public_ip_address }}
            StrictHostKeyChecking no
            UserKnownHostsFile ~/.ssh/known_hosts_amz        
    loop: "{{ instance.instances }}"
    when: item.public_ip_address is defined
  tags: ['never', 'destroy']

Use of Ansible Tags and Blocks

Blocks are used in playbooks to group related tasks together.

I’m also using tags to specify which tasks should be run based on creation or destruction.

Tags are used like this:

# Create the server
ansible-playbook ec2-manage.yml --tags=create

# Destroy the server
ansible-playbook ec2-manage.yml --tags=destroy

Ansible has the special tags “always” and “never”, as well. “never” will keep a task from running unless you call it specifically by tag, as I’m doing here.

I’ll probably reorganize, retag, or even split the playbook in the future, depending on how I decide to use it.

The Full Playbook

---
- name: Manage EC2 instances
  hosts: x270
  # gather_facts: false
  vars:
    project: test
    unit: server
    aws_region: us-east-1
    availability_zone: us-east-1a
    instance_type: t2.micro
    # Ubuntu 22.04
    image_id: ami-007855ac798b5175e
    inventory_host_groups: "{{ unit }}"
    host_name: "{{ project }}_{{ unit }}"
    ssh_key_name: "{{ project }}_{{ unit }}"
    ssh_key_path_local: "~/.ssh/{{ ssh_key_name }}.pem"
    backup_ssh_config: false
    print_instance_info: false

  tasks:

    - name: Get info block
      block:
        - name: Get running instance info
          ec2_instance_info:
          register: ec2info
        - name: Print instance info
          debug: var="ec2info.instances"
          when: print_instance_info
      tags: ['always']

    - name: Create an EC2 key
      block:
        - name: Create the EC2 key
          amazon.aws.ec2_key:
            name: "{{ ssh_key_name }}"
            region: "{{ aws_region }}"
            force: true
          register: ec2key
        - name: Save the private key locally
          ansible.builtin.copy:
            content: "{{ ec2key.key.private_key }}"
            dest: "{{ ssh_key_path_local }}"
            mode: 0600
          when: ec2key.changed
      tags: ['create']

    - name: Terminate EC2 Block
      block:
      - name: Get EC2 instance info
        amazon.aws.ec2_instance_info:
          filters:
            "tag:Name": "{{ host_name }}"
        register: instance
      - name: Terminate EC2 instances
        amazon.aws.ec2_instance:
          state: terminated
          name: "{{ ssh_key_name }}"
        register: terminated_instances
      - name: Remove the EC2 key
        amazon.aws.ec2_key:
          name: "{{ ssh_key_name }}"
          state: absent
      - name: Remove security group
        amazon.aws.ec2_security_group:
          name: "{{ host_name }} SSH"
          state: absent
      - name: Delete local ssh key
        ansible.builtin.file:
          path: "{{ ssh_key_path_local }}"
          state: absent
      # TODO Remove known hosts entry
      - name: Update local ansible hosts file
        ansible.builtin.lineinfile:
          path: ./hosts
          search_string: "{{ host_name }}"
          state: absent
      - name: Remove block from local .ssh/config
        ansible.builtin.blockinfile:
          path: ~/.ssh/config
          state: absent
          block: |
            Host {{ host_name }}
                IdentityFile {{ ssh_key_path_local }}
                IdentitiesOnly yes
                User ubuntu
                HostName {{ item.public_ip_address }}
                StrictHostKeyChecking no
                UserKnownHostsFile ~/.ssh/known_hosts_amz            
        loop: "{{ instance.instances }}"
        when: item.public_ip_address is defined
      tags: ['never', 'destroy']

    - name: Create EC2 ssh security group
      block:
      - name: Create security group
        amazon.aws.ec2_security_group:
          name: "{{ host_name }} SSH"
          description: SSH access for instance
          region: "{{ aws_region }}"
          rules:
            - proto: tcp
              ports: 22
              cidr_ip: 0.0.0.0/0
              rule_desc: allow all ssh on port 22
        register: security_group
      tags: ['never', 'create']

    - name: Create EC2 Block
      block:
      - name: Create EC2 instances
        amazon.aws.ec2_instance:
          name: "{{ ssh_key_name }}"
          region: "{{ aws_region }}"
          availability_zone: "{{ availability_zone }}"
          key_name: "{{ ssh_key_name }}"
          instance_type: "{{ instance_type }}"
          image_id: "{{ image_id }}"
          user_data: |
            #!/bin/bash
            sudo apt update && sudo apt upgrade
            sudo awk '/root\s+ALL=\(ALL:ALL\) ALL/ {print; print "ubuntu ALL=(ALL) NOPASSWD:ALL"; next}1' /etc/sudoers | sudo tee /etc/sudoers            
          security_groups: ['default', "{{ security_group.group_id }}"]
          network:
            assign_public_ip: true
        register: ec2

      - name: Wait for ssh to come up
        local_action:
          module: wait_for
          host: "{{ item.public_ip_address }}"
          port: 22
          delay: 10
          timeout: 120
        loop: "{{ ec2.instances }}"

      - name: Update local .ssh/config
        ansible.builtin.blockinfile:
          path: ~/.ssh/config
          backup: "{{ backup_ssh_config }}"
          block: |
            Host {{ host_name }}
                IdentityFile {{ ssh_key_path_local }}
                IdentitiesOnly yes
                User ubuntu
                HostName {{ item.public_ip_address }}
                StrictHostKeyChecking no
                UserKnownHostsFile ~/.ssh/known_hosts_amz            
        loop: "{{ ec2.instances }}"
        # TODO
        # when: ec2key.changed

      - name: Ensure there is a group line for our instances in hosts file
        ansible.builtin.lineinfile:
          path: ./hosts
          line: "[{{ inventory_host_groups }}]"
          state: present

      - name: Update local ansible hosts file
        ansible.builtin.lineinfile:
          path: ./hosts
          line: "{{ host_name }}"
          insertafter: "[{{ inventory_host_groups }}]"
          state: present

      - name: print
        debug: var="item.public_ip_address"
        loop: "{{ ec2.instances }}"

      tags: ['never', 'create']

Drawbacks

There are a few downsides to this playbook:

Certain tasks aren’t set up to handle multiple servers.
Most of the variables should be configured elsewhere so that the playbook can be used for multiple types of servers in different regions.
The known_hosts file never gets cleaned up.

Security Issues

There is some basic hardening that this setup is missing:

SSH isn’t hardened.
The security group allows SSH access from any IP address.
StrictHostKeyChecking is disabled in the ~/.ssh/config file.
Passwordless sudo is enabled for the “ubuntu” user.
The instance storage is not encrypted.