How to get the same Private Static IP, always attached to an EC2 instance in Auto scaling group, using lifecycle hook

When you launch an EC2 instance, a primary private IPv4 address for the instance gets allocated to it. An instance receives a primary private IP address from the IPv4 address range of the subnet. A private IPv4 address, remains associated with the network interface when the instance is stopped and started, or hibernated and started, and is released when the instance is terminated and new instance gets new private IP address.

What if, your use case requires, retaining the same specific Private IP address for an EC2 instance, even after it is released on termination ? e.g. you need specific static private IP 10.8.16.34/32 always attached to an EC2 instance, irrespective of termination.

One of the solutions is to use the Elastic IP address but since EIP gives the public IP address, it would not work if you are building infrastructure in an organisation where public cloud platform is very restricted and controlled such as Bank, due to its reachability from Internet.

This solution utilizes Secondary ENI (Elastic Network Interface) provided by AWS along with lifecycle hook, Lambda and parameter store.

solution diagram

If you do not know how the lifecycle hook works then please refer the link here.

Auto Scaling instance states

The main challenge in designing this solution was, making sure the Secondary ENI gets disassociated from the terminated EC2 instance and gets associated with the new EC2 instance automatically, even in the event of multiple EC2 termination.

Another challenge was the management of the ENIs which was solved by using the Parameter store.

Now lets assume EC2 instances running in an auto scaling group, where desired capacity in the auto scaling group is 3. As a pre-requisite, all the EC2 instances have been given tag key:name=value:peer1, peer2 etc and a secondary ENI has been associated with each of them.

Here is how the solutions works:

1. Lets consider that EC2 instance tagged as ‘peer 1’ goes down or say gets terminated.
2. The lifecycle hook gets triggered and EC2 Auto scaling submits an event for a lifecycle action to EventBridge. (Notice the ‘Instance terminate’ is selected in ‘Lifecycle transition’ field.)

EC2 auto scaling lifecycle hook — terminate

3. Next step is to receiving the event in the EventBridge, which triggers a lambda function which does the job to detach the secondary ENI from the terminating EC2 instance and save the tag & 2nd ENI into the parameter store.

4. Lambda function is very simple which reads the event data, fetches the tag key/value, secondary ENI value, put it in a parameter store and completes the lifecycle action.

5. When auto scaling spins up a new EC2 instance, (it goes into the pending:wait state) and another lifecycle hook gets triggered and EC2 Auto scaling submits an event for a lifecycle action to EventBridge. (This time notice the ‘Instance launch’ is selected in ‘Lifecycle transition’ field.)

EC2 auto scaling lifecycle hook — launch

6. EventBridge rule to receive the launch lifecycle hook event which invokes another lambda function which fetches the saved tag key/value & secondary ENI from the parameter store and attaches it to the newly created EC2 instance and completes the lifecycle hook action.

In case of multiple EC2 termination, the launch lambda function fetches the tag & secondary ENI pair which were stored first in the parameter store.. think of it as FIFO queue :). lambda function also deletes the tag & secondary ENI pair from the parameter store once it successfully associates them with EC2 instance.

By implementing this solution you can not only automatically disassociate and reattach the secondary ENI to an EC2 instance in the auto scaling group but also use the same secondary ENI.

Hope this blog would have helped you!

Happy Learning!