Event-Driven serverless architecture to automatically remediate security findings

Security remediation framework — Architecture

We all love to design, build and deploy on AWS. But as the cloud adoption is growing, the security concerns in and around the cloud is also growing. We see and hear about the security breaches and incidents very often now a days.

Having spent a good amount of time in migrating, building and deploying applications on AWS for multiple customers, I can say that organizations take their cloud security very seriously, specially the financial firms. I’ve seen the most strict and controlled AWS environments, the best implementations of DevOps pipelines, maturity level of firms to built in their internal processes into the AWS environment and the security of cloud resources through extensive preventive and detective controls.

If you are owing an AWS platform in your firm and looking for a fully automated security solution or do not want to spend huge money in buying the managed security solution from vendors then this blog will help you implement the security solution in-house no matter how big or small your firm is.

In this bog, I’ll walk you through the solution, based on the event-driven architecture, to detect and remediate the security findings automatically using AWS managed security services.

The logic behind this solution is to collect the security findings at central place, so making it easy to detect, capture and respond; irrespective of the number of AWS Accounts being used.

Next question is how to implement it?

In AWS, there are services which work on master-member model to centralize logs/findings/events, few example are AWS Security Hub, AWS Guard Duty, Config etc.

1. Amazon GuardDuty — a threat detection service which continuously monitors for malicious activity and capable of analyzing tens of billions of events across multiple AWS data sources, such as AWS CloudTrail event logs, Amazon Virtual Private Cloud (VPC) flow Logs, and DNS query logs.
2. AWS Security Hub — a cloud security posture management service that performs automated, continuous security best practice checks against your AWS resources.
3. AWS Config — This service enables you to assess, audit, and evaluate the configurations of your AWS resources. Config continuously monitors and records your AWS resource configurations and allows you to automate the evaluation of recorded configurations against desired configurations.

Let’s take an assumption that there are 10 AWS Accounts, where the workloads are running, let’s call them ‘AWS Solution' accounts

We then need one AWS account which will work as a central place to collect the security findings and events, let’s call it 'Core-compliance' account.

We know that every action that is performed in AWS is basically an API call and every action emits an event — a signal that a system’s state has changed. These events get collected in the event bus in Amazon Event Bridge service.

I have enabled AWS GuardDuty as member, in each of the AWS Solution accounts where are Master can either be your core-compliance AWS Account or your root/master AWS account, because AWS Security Hub will used to detect the findings from Amazon GuardDuty.

Since, Security Hub is integrated with AWS services like IAM Access Analyzer, Macie, Firewall Manager, Amazon Inspector etc. I have enabled Security Hub as member in each of the AWS Solution accounts and as Master in the central core-compliance AWS Account, hence with this setup, all the Security Hub findings will be pushed & collected at a central place.

I have also enabled IAM Access Analyzer, which helps identifying the resources in organization and accounts, such as Amazon S3 buckets or IAM roles, shared with an external entity. Findings from IAM Access Analyzer will also be sent to Security Hub member which eventually goes to master Security Hub.

You might be aware that Security Hub processes these findings using a standard findings format called the AWS Security Finding Format (ASFF).

To enable AWS Config, I have used the AWS Config Conformance pack. The Conformance pack is a collection of AWS Config rules and remediation actions that can be easily deployed as a single entity in an account and a Region or across an organization in AWS Organizations. More details are available here. AWS provides ready-made sample conformance pack which can be deployed easily.

AWS Config — conformance pack

Security Hub doesn’t get findings from AWS Config by default, hence a solution needs to be in place to send Config rule evaluations as findings into Security Hub, use the official AWS blog here, this will import AWS Config rules evaluations as findings in Security Hub in the ASFF format which Security Hub uses.

Now, I have all the findings from all 10 AWS Accounts into the member Security Hub and eventually into the Security Hub (Master).

By default, Security Hub automatically sends all new findings and all updates to existing findings to EventBridge as Security Hub Findings — Imported events. Please refer here for more details

The security findings do not get detected instantly, it take few minutes to reflect in the services like GuardDuty, Security Hub or AWS Config but I wanted to capture some high severity events in near real time such as someone tries to attach internet gateway in AWS Account, or someone has applied unrestricted S3 bucket policy, or someone is adding 0.0.0.0/0 as Inbound rule in a security group.

As I said earlier, every actions in AWS is an API call and every action generates events, hence a custom event pattern is created to capture all the events from the solution account with target set as the ARN of the default event bus of core-compliance AWS Account. These events takes no time to be available in Event Bridge.

event pattern to capture all Events of solution AWS account

Event rule target — Event bus of core-compliance account

Now, I have all the findings generating out of different Security services, all the events generating from actions from all AWS Solution accounts, at one place.

Next step is, how do we capture these events? The answer is using the Event patterns in Event Bridge service e.g. to capture the event of an action when someone adds internet gateway to the VPC; for that the custom event pattern looks like this:

custom event pattern for capturing event from action

custom event pattern for capturing event from Security Hub findings

Now that the event is captured, it is now time to take action. I have used a fleet of lambda functions to do this job - code that reads the event, retrieves the AWS account details from it, then gains access into the AWS account and finally takes action to do what it is supposed to do; in above scenario detaching the internet gateway from the VPC.

Wondering how does lambda gained access to the impacted AWS Account? Answer is by assuming an IAM Role, which is already deployed in each of the AWS Solution accounts, for Lambda to assume.

Now the last step is, to notify the AWS account owner or a contacts DL, if configured for an account, which can be used to send the Email notification to, regarding the remediation which took place in their AWS Account. The best way to send notifications is by using AWS SNS service.

With every custom event pattern along with Lambda as a target, SNS topic as a target which sends the notification to the owner/DL.

That’s all !! It’s done! :)

You have a security framework in place which monitors AWS accounts for any unwanted activities, takes action and notifies you that too automatically. A complete peace of mind ☺️ isn’t it?

You can improve this framework, by introducing more services such as Amazon Detective, Macie, Audit Manager and also integrate it with change management tools such as JIRA, Service Now etc, or deploy the framework in all AWS regions, to make it more powerful.

Hope this blog will help you in building the security framework which automatically remediates the security findings. Feel free to put your questions in the comment section, I’ll try my best to answer those at the earliest. Also, do clap if you find this blog useful.

Happy Learning!