TL;DR

Writing a custom resource for this purpose is technically a big challenge and might not be worth the effort. Doing it through AWS SDK or AWS CLI is possible.

Jump to the script that does this using AWS CLI.

Backstory

Recently at Thundra, we rebuilt part of our infrastructure from scratch using AWS CDK. With this new infrastructure, we minimized the parts that we did manually before, simplified and divided our resources into multiple stacks with a lot of nesting.

The motivation behind this was to remove the current technical debts in our core infrastructure and automate our environment setups. We’ve been using AWS CDK for a while now and we know our way around it. We know the pitfalls; we know what’s possible and the workarounds.

Sadly, we couldn’t automate everything through AWS CDK.

Problem

Let us assume we have Stack A, which creates a VPC and an instance. Then we have Stack B, which has some resources such as RDS in a separate VPC and a Security Group that’s attached to this RDS instance.

The third stack, Stack C, is the peering stack which enables the resources in A to access the ones in B and vice versa.

Suppose that the resources in B are in an isolated subnet and the only way to access them is through the Security Groups within its VPC. With Stack C, we create a peering between VPC A and B. Then, it’s possible to create a new Security Group in A and add an inbound rule to the Security Group in B to accept requests from this new Security Group. If we want our instance in A to have access to the RDS in B, this new Security Group should also be attached to the instance.

Unfortunately, you can’t do EC2 lookups in AWS CDK. At least not yet. It’s possible to describe and modify EC2 instances through AWS SDK and CLI, which we’ll come to it later in this post.

Possible Solution

So, we thought maybe we can write a custom resource to find the targetted instance and modify it. It’s certainly possible, though it’s not a trivial task. The real problem isn’t the capability, but the behaviour of both Security Groups and EC2 Instances when an update or delete action affects them.

Some actions on Security Groups will invoke replacement on them, like updating its description or name. Similarly, changing the instance type on an instance will replace the instance for obvious reasons.

Coming back to the custom resource; even if we figured out all the difficulties that come with the CloudFormation events, it’s still won’t be possible to find a suitable place to put the resource in our codebase.

Remember that the order our stack applies is like so;

1. Stack A
   • VPC A
   • EC2 Instance
2. Stack B
   • VPC B
   • RDS
   • RDS Security Group
3. Stack C
   • VPC Peering between A and B

Knowing that the new Security Group needs the RDS Security Group and the VPC Peering, this supposed custom resource must be in C. However, if at any point in the future an update on Stack A triggers replacement on the EC2 instance, the newly created instance won’t have the Security Groups attached to it. Triggering C again won’t do anything. Since, from its point of view, nothing has changed.

If any update on C triggers replacement on the Security Group, the custom resource must detach the Security Group first. Figuring the side-effects of these events on a custom resource is quite difficult if it is all possible.

The Solution

At this point, we have accepted our fate and made peace with just running an extra script that updates the targetted EC2 Instance when necessary.

Following script relies on AWS CLI. However, similar capabilities are available in AWS SDK as well. The script itself has room for improvement, of course.

For the sake of simplicity, I will just give it as is for now.

#!/bin/bash

export INSTANCE_NAME=<TARGET_INSTANCE_NAME>
export NEW_SG_NAME=<NEW_SG_NAME>
​
# This will get the instance ID of the targetted
# EC2 Instance that's currently running.
#
# The filter is up to you. Change it as you wish.
#
export TARGET_INSTANCE_ID=$(aws ec2 describe-instances \
    --region $REGION \
    --filters \
        Name=instance-state-name,Values=running \
        Name=tag:Name,Values=$INSTANCE_NAME \
    --query "Reservations[*].Instances[*].InstanceId" \
    --output text)
​
# This will get the Security Groups that are currently
# attached to the instance.
#
export TARGET_INSTANCE_SG_GROUPS=$(aws ec2 describe-instances \
    --region $REGION \
    --instance-ids $TARGET_INSTANCE_ID \
    --query "Reservations[*].Instances[*].SecurityGroups[*].GroupId" \
    --output text)
​
# This will get the new Security Group we created in
# Stack C.
#
# It will again find it via its name. Change it as you wish.
#
export NEW_SG_ID=$(aws ec2 describe-security-groups \
    --region $REGION \
    --filters \
        Name=tag:Name,Values=$NEW_SG_NAME \
    --query "SecurityGroups[*].GroupId" \
    --output text)
​
# Following if block will check if the new Security Group
# is already attached to the instance.
#
# If not​, it'll append the $NEW_SG_ID to $TARGET_INSTANCE_SG_GROUPS
# ​
if [[ ! ${TARGET_INSTANCE_SG_GROUPS[*]} =~ $NEW_SG_ID ]]; then
    echo "Not Found $NEW_SG_ID in $TARGET_INSTANCE_SG_GROUPS"
​
    export TARGET_INSTANCE_SG_GROUPS=(${TARGET_INSTANCE_SG_GROUPS[*]} $NEW_SG_ID)
fi
​
echo "Updating security groups of Instance:$TARGET_INSTANCE_ID with ${TARGET_INSTANCE_SG_GROUPS[*]}"
​
aws ec2 modify-instance-attribute \
    --region $REGION \
    --instance-id $TARGET_INSTANCE_ID \
    --groups ${TARGET_INSTANCE_SG_GROUPS[*]}
​
echo "Done"

That’s all for now. Let me know if you have something on your mind!