Centralized Storage with Amazon EFS in an Autoscaling Environment with Ansible
In today’s dynamic cloud infrastructure, the need to scale resources seamlessly to handle varying workloads is paramount. Autoscaling groups in AWS provide a solution to dynamically adjust the number of EC2 instances based on demand. However, managing the storage of ephemeral instances can pose challenges, especially when dealing with logs and other persistent files.
The Power of Autoscaling Groups
To address the demands of growing traffic in applications and businesses, the implementation of autoscaling groups becomes crucial. Autoscaling allows you to define scaling policies based on metrics like CPU utilization or network traffic, ensuring your application can handle increased load without manual intervention.
Building a Golden Image with Packer and Ansible
My choice for implementing autoscaling of EC2 instances involves creating a golden image using Packer. Packer, in combination with Ansible as the provisioner, allows me to build an Amazon Machine Image (AMI) that contains all the necessary requirements and dependencies for my application. This golden image serves as the foundation for the instances launched by the autoscaling group.
Here’s a snippet of the Packer template (template.json):
// ... (Previous Packer configuration)
"provisioners": [
{
"type": "shell",
"script": "/tmp/init.sh"
},
{
"type": "shell",
"inline": [
"sleep 30",
"df -h",
"lsblk -o NAME,FSTYPE /dev/nvme1n1"
]
},
{
"type": "ansible",
"galaxy_file": "{{user `ansible_dir`}}/ansible/requirements.yml",
"playbook_file": "{{user `ansible_dir`}}/ansible/ansible/playbooks/configure.yaml",
"extra_arguments": ["-v","--extra-vars", "artifact_path={{ user `project_path` }} EFS_DNS_NAME={{ user `efs_dns_name` }} EFS_MOUNT_TARGET_IP={{ user `efs_mount_target_ip` }}"],
"user": "{{user `ansible_remote_user`}}"
}
],
// ... (Subsequent Packer configuration)
In this template, I’m using Ansible to configure the EC2 instance with the necessary settings. Additionally, I’m passing variables related to the Amazon EFS setup, such as DNS name and mount target IP, to the Ansible playbook.
Centralized Storage with Amazon EFS
When running multiple dynamic instances from an auto-scaling group, the logs generated in each instance are usually stored in the local storage of that instance. However, with auto-scaling, instances may come and go, leading to potential data loss. This is where a centralized storage solution becomes essential.
In scenarios where central logging solutions such as ELK, cloud-provided options like CloudWatch, or third-party solutions like Datadog are not utilized, leveraging Amazon Elastic File System (EFS) for log storage emerges as a cost-effective and efficient alternative. By storing logs in EFS, we create a shared repository accessible by dynamic instances within an autoscaling environment. This not only ensures persistence across instances but also significantly reduces costs compared to alternative solutions. Moreover, the simplicity of utilizing bash tools for parsing logs in a Linux server further enhances the practicality of this implementation, making it an attractive choice for environments where centralized logging platforms might be overkill or cost-prohibitive.
Amazon EFS to the Rescue
Amazon Elastic File System (EFS) provides scalable and shared file storage that can be mounted on multiple EC2 instances. This makes EFS an ideal solution for storing logs and other files that need to persist across instances.
Ansible Role for EFS Configuration
To facilitate the configuration of EFS on EC2 instances, I’ve created an Ansible role named "efs". This role ensures the necessary dependencies are installed, clones the amazon-efs-utils from GitHub, builds the required deb packages, installs them on the instances, and finally mounts the EFS volume.
Here’s a snippet of the Ansible role (tasks/main.yaml):
// … (Previous Ansible tasks)
- name: Update repositories for EFS packages
apt:
update_cache: yes
upgrade: dist
state: latest
- name: Make sure the efs dependency packages are installed (Debian/Ubuntu)
apt:
pkg: "{{item}}"
state: present
update_cache: yes
with_items:
- git
- binutils
when: ansible_distribution in [ "Debian", "Ubuntu" ]
- name: Clone amazon-efs-utils from GitHub
ansible.builtin.git:
repo: "https://github.com/aws/efs-utils"
dest: "/tmp/efs-utils"
update: true
version: "master"
- name: Build deb packages from the aws/efs-utils code
ansible.builtin.command: ./build-deb.sh
args:
chdir: /tmp/efs-utils
creates: /tmp/efs-utils/build
- name: Find all aws/efs-utils deb packages that were built
ansible.builtin.find:
paths: /tmp/efs-utils/build
patterns: "amazon-efs-utils*.deb"
register: find_result
- name: Install aws/efs-utils deb packages
ansible.builtin.apt:
deb: "{{ item }}"
loop: "{{ find_result | json_query('files[*].path') }}"
- name: Create mount point directory
file:
path: /efslogs
state: directory
- name: Mount EFS volume
mount:
path: /efslogs
src: "{{EFS_DNS_NAME}}:/"
fstype: efs
opts: "tls,mounttargetip={{EFS_MOUNT_TARGET_IP}}"
state: mounted
In this role, I’m using Ansible to perform tasks such as updating repositories, installing dependencies, cloning the amazon-efs-utils from GitHub, building and installing deb packages, creating a mount point directory, and finally, mounting the EFS volume.
The CodeBuild builder instance should be able to contact the mount target created during EFS provisioning. In most cases, public subnets are used for the builder instance in CodeBuild, while the service nodes, which run in an autoscaling group, are placed in private subnets. For this scenario, it is recommended to create mount targets in EFS exclusively for private subnets. Due to the builder instance in CodeBuild being unable to reach the mount target, the Ansible task for mounting the EFS volume may not work as intended. In such cases, it is advisable to exclude that task from the Ansible role and instead incorporate the mount script into the EC2 init script.
# Mount EFS
findmnt --mountpoint /efs
if [ $? -ne 0 ]; then
mount -t efs -o tls,mounttargetip=${efs_mount_target_ip} ${efs_filesystem_id} /efs/
else
echo "EFS filesystem is already mounted"
fi
Terraform Orchestration for Amazon EFS Provisioning
In conjunction with the robust Packer and Ansible setup for creating a golden EC2 image, the Terraform implementation adds another layer of sophistication to infrastructure orchestration. This Terraform script focuses on provisioning Amazon Elastic File System (EFS), offering a centralized and scalable solution for shared file storage. The script elegantly encapsulates various aspects of EFS configuration, including file system parameters, lifecycle policies, security groups, access points, and replication configurations. The modular structure of the script ensures adaptability, allowing seamless integration with existing infrastructure components. Leveraging the Terraform VPC module further enhances the overall network layout, establishing a resilient foundation for the EFS deployment. The inclusion of a customer-managed KMS key ensures data security through encryption. Altogether, this Terraform implementation, when combined with the earlier Packer and Ansible components, presents a comprehensive and automated approach to deploying, configuring, and managing EFS in an AWS environment.
Here’s a snippet of the Terraform:
provider "aws" {
region = local.region
}
locals {
region = "ap-south-1"
name = "ex-${basename(path.cwd)}"
azs = slice(data.aws_availability_zones.available.names, 0, 3)
tags = {
Name = local.name
Example = local.name
Repository = "https://github.com/terraform-aws-modules/terraform-aws-efs"
}
}
data "aws_availability_zones" "available" {}
data "aws_caller_identity" "current" {}
################################################################################
# EFS Module
################################################################################
module "efs" {
source = "../.."
# File system
name = local.name
creation_token = local.name
encrypted = true
kms_key_arn = module.kms.key_arn
performance_mode = "maxIO"
throughput_mode = "provisioned"
provisioned_throughput_in_mibps = 256
lifecycle_policy = {
transition_to_ia = "AFTER_30_DAYS"
transition_to_primary_storage_class = "AFTER_1_ACCESS"
}
# File system policy
attach_policy = true
bypass_policy_lockout_safety_check = false
policy_statements = [
{
sid = "Example"
actions = ["elasticfilesystem:ClientMount"]
principals = [
{
type = "AWS"
identifiers = [data.aws_caller_identity.current.arn]
}
]
}
]
# Mount targets / security group
mount_targets = { for k, v in zipmap(local.azs, module.vpc.private_subnets) : k => { subnet_id = v } }
security_group_description = "Example EFS security group"
security_group_vpc_id = module.vpc.vpc_id
security_group_rules = {
vpc = {
# relying on the defaults provdied for EFS/NFS (2049/TCP + ingress)
description = "NFS ingress from VPC private subnets"
cidr_blocks = module.vpc.private_subnets_cidr_blocks
}
}
# Access point(s)
access_points = {
posix_example = {
name = "posix-example"
posix_user = {
gid = 1001
uid = 1001
secondary_gids = [1002]
}
tags = {
Additionl = "yes"
}
}
root_example = {
root_directory = {
path = "/example"
creation_info = {
owner_gid = 1001
owner_uid = 1001
permissions = "755"
}
}
}
}
# Backup policy
enable_backup_policy = true
# Replication configuration
create_replication_configuration = true
replication_configuration_destination = {
region = "eu-west-2"
}
tags = local.tags
}
################################################################################
# Supporting Resources
################################################################################
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
version = "~> 5.0"
name = local.name
cidr = "10.99.0.0/18"
azs = local.azs
public_subnets = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
private_subnets = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"]
enable_nat_gateway = false
single_nat_gateway = true
tags = local.tags
}
module "kms" {
source = "terraform-aws-modules/kms/aws"
version = "~> 1.0"
aliases = ["efs/${local.name}"]
description = "EFS customer managed key"
enable_default_policy = true
# For example use only
deletion_window_in_days = 7
tags = local.tags
}
Putting It All Together
Golden Image Creation: The process starts with creating a golden image using Packer. This image includes the necessary configurations and dependencies for the application.
Autoscaling with Terraform: Terraform is then used to deploy the AMI to the autoscaling group. This ensures that the environment scales seamlessly based on specified criteria, meeting the demands of varying traffic.
Centralized Storage with EFS: Ephemeral EC2 instances often result in data loss. By employing Amazon EFS, storage is centralized, providing persistence for logs and other essential files. This is particularly valuable when central logging solutions like ELK or cloud-provided solutions are not used.
Conclusion
In conclusion, the implemented solution offers a cost-effective and scalable approach to managing EC2 instances in an autoscaling group. The combination of Packer, Ansible, Terraform, and Amazon EFS provides automation, ease of deployment, and persistent storage, addressing the challenges associated with dynamic environments.
By centralizing storage with EFS, data continuity is ensured even when instances are dynamically added or removed. This implementation serves as a robust foundation for handling varying workloads, supporting the growth of applications, and optimizing infrastructure resources.