Invoking Private REST APIs (Amazon API Gateway) with custom domain name and SSL, from On-premise
Introduction
Lot of AWS customers are having hybrid deployment where their systems running at their data center need to access APIs running in AWS environment over Direct Connect/VPN. Enabling Private APIs running in AWS environment and accessing it using custom domain names, from systems running in data center, can help customers building the secure Hybrid environment and run their API-powered applications in a seamless manner specially when paradigm is shifting towards building microservices application design pattern.
One of the customers I worked with is a worldwide bank that has been migrating its on-premise workloads to AWS but due to some technical limitations, their Enterprise API gateway would need to remain at on-premise and keep Authenticating and Authorizing the API requests (with SAML token) for APIs hosted on AWS serverless compute via Amazon API Gateway Private Endpoints. The hybrid environment setup required the use of custom domain name (such as abcbank.com) for private REST APIs, so that these API URLs can be exposed at their Enterprise API gateway.
At the time of writing this article, and as per official AWS documentation, Custom domain names are not supported for private APIs.
Solution overview
For demo perspective, I have simplified the solution; I’m using the public certificate from ACM, Public R53 hosted zone and Simulated On-premises environment with VPC and VPC Peering.
Following are the steps taken to build the solution:
- Request/Import private certificate(s) in Amazon Certificate Manager (I’m using Public certs for demo)
- Create Interface VPC Endpoint for execute-api with private DNS enabled
- Create APIs in either Lambda/ECS/EKS (I’m using Lambda for demo)
- Create API Gateway with Private Endpoint
a. Create API integration
b. Create custom domain names with certificates and configure API mapping
c. Create API resource policy to allow traffic from VPC endpoint only
d. Deploy APIs
5. Create Network Load Balancer or Application Load balancer with private certificate and target to the VPC endpoint ENI (I’m using Network Load balancer for demo)
6. Create Amazon Route 53 Alias records in the Private hosted zone with traffic routing to NLB (I’m using Public hosted zone for demo)
7. Simulated On-premises environment with VPC and VPC Peering (Additional step for demo purpose only)
High-level architecture of the implemented solution
I have taken the use case to showcase the real-world scenario where customer may have multiple APIs or single API with multiple methods and resources.
Getting started
To follow along with this post, you must have:
1. Already have the VPN or Direct Connect connection between your On-Premise and AWS environment
2. AWS account having an application VPC with connectivity to On-premise
3. Private subnets for deploying the workloads & infrastructure
4. A registered domain name in order to set up custom domain names for your APIs
For full disclosure, I have already setup Two VPCs — Default VPC for deploying workloads & Simulated On-premise and have created the VPC Peering connection between them (Step# 7):
Step 1: Request/Import certificate in AWS Certificate Manager:
I already have a Public certificate requested from Amazon. If you need guidance about how to Import/request Public or Private certificate via AWS Certificate Manager, I’d suggest Issuing and Managing Certificates
Step 2: Create Interface VPC Endpoint for API Gateway:
API Gateway private endpoints are made possible via AWS PrivateLink interface VPC endpoints. Interface endpoints work by creating elastic network interfaces in subnets that you define inside your VPC. If you like to understand more on API Gateway evolution, do refer Introducing Amazon API Gateway Private Endpoints
1. Open the Amazon VPC console
2. In the left navigation pane, choose Endpoints, Create Endpoint
3. For Service category, keep it set to “AWS Services”
4. For Service Name, set it to “com.amazonaws.{region}.execute-api”.
5. For VPC, select the Application VPC where you want to deploy the APIs. APIs can only be accessed using the deployed ENIs in the VPC you choose.
6. For Subnets, select the private subnets from application VPC. Make sure that you provide multiple interfaces in your VPC i.e. endpoint in each subnet in each Availability Zone for which the VPC is configured.
7. For Enable Private DNS Name, keep it checked as Enabled for this endpoint.
8. Attach the security group which only allows the desired traffic to invoke the APIs. If you are planning to use the Application Load Balancer, then this security group should only allow specific port & protocol from ALB security group.
9. For Resource policy, keep the default
10. Choose Create Endpoint
11. VPC Endpoint will be created with pending state, wait for few minutes until it shows the status as ‘available’. I’d suggest to refer Interface VPC endpoints (AWS PrivateLink)
In the Subnets tab, you’ll get the private IP address of the ENIs, if you have chosen multiple subnet (one in each AZ) then you should see IP address for each subnet. I’ll use this IP address as target of ELB.
Also note the Endpoint ID, it will be used while creating the Private API in API Gateway.
Security group used by VPC Interface endpoint, allows Port 80 & 443 traffic from application VPC and Simulated On-premise only -
Step 3: Creating APIs:
For simplicity, I have created very basic Lambda functions in Python runtime environment. I will use three Lambdas for two APIs.
These Lambda functions are created in the subnet of application VPC where the API Gateway ENIs are deployed. Security group used by Lambdas, allows Port 80 & 443 traffic from application VPC only.
1. Lambda 1 will work as 1st resource of First API in API Gateway
2. Lambda 2 will work as 2nd resource of First API in API Gateway
3. Lambda 3 will work as Second API in API Gateway
You can deploy your APIs in ECS or EKS with Fargate/EC2 also, depending on your requirements and use case.
Step 4: Creating API Gateway with Private Endpoint
4a. I’ll create two Private REST APIs in API Gateway. One API with two resources and One API with single resource. Refer Creating a private API in Amazon API Gateway
1. Go to API Gateway service
2. Choose REST API Private and click on Build
3. Select ‘New API’ checkbox –
a. Enter your API name
b. Enter description (Optional)
c. Choose the Endpoint Type as ‘Private’
d. In the VPC Endpoint IDs textbox, enter the VPC ID created earlier (refer step# 11)
e. Click on Create API button
4. Under Resources, you’ll see nothing but /. This is the root-level resource, which corresponds to the base path URL for your API (https://<API id>. execute-api.<region>.amazonaws.com/{stage-name}).
From the Actions dropdown menu, choose Create Method.
5. Select the method type for your API. Here I’m going to get the value from the Lambda hence I’m using GET method.
6. In the / — GET — Setup pane, for Integration type, choose Lambda Function. If you have not created your APIs in Lambda, then choose the appropriate Integration type. I’d suggest to refer Choose an API Gateway API integration type
7. Choose Use Lambda proxy integration.
8. For Lambda Region, choose the Region where you created your Lambda function.
9. In the Lambda Function field, type any character and then choose the appropriate Lambda (or whatever name you gave the function that you created in the previous step) from the dropdown menu. (If the dropdown menu doesn’t appear, delete the character you just typed to make the dropdown menu appear.) Leave Use Default Timeout checked. Choose Save to save your choice.
I have used the First Lambda function ‘Hello-lambda1’ which is the 1st resource of our First API.
10. When the Add Permission to Lambda Function popup appears (saying “You are about to give API Gateway permission to invoke your Lambda function…”), choose OK to grant API Gateway that permission.
11. Now you’ll see a / — GET — Method Execution pane:
We are now good with our First resource for our 1st API. I need to create a child resource for this API where I’ll choose our Second Lambda function ‘Hello-lambda2’.
12. From the Actions dropdown menu, choose Create Resource. Name the resource and click on Create Resource
13. Repeat step# 7–11 to integrate the second Lambda function.
14. Now you’ll see a /test-resource — GET — Method Execution pane:
Till this point we have built our First API — ‘Pvt-lambda-test-api’ with two API resources — one is the default resource integrated with ‘Hello-lambda1’ lambda function and second is the child resource integrated with ‘Hello-lambda2’ lambda function.
4b. Create Resource Policy
Before I deploy the APIs, I need to configure access control to this private API using a Resource Policy. Access can be controlled by IAM condition elements, including conditions on AWS account, Source VPC, VPC Endpoints (Private API), and/or IP range.
Learn more Controlling access to an API with API Gateway resource policies
For demo perspective, are using a simple Resource policy which ensure that the API is accessible only through the VPC Interface Endpoint –
4c. Deploy your REST API
Once you complete above step, you’ve created an API, but you can’t actually use it yet. This is because it needs to be deployed.
1. From the Actions dropdown menu, choose Deploy API
2. From the Deployment stage dropdown menu, choose [New Stage]
3. For Stage name, enter the stage name (I used demo)
4. Choose Deploy
In the Stage Editor, note the Invoke URL at the top. It should be in this format: (https://nwnx6564r0.execute-api.us-east-1.amazonaws.com/demo). If you choose the Invoke URL, it will open a new browser tab with that URL since but you’ll NOT any response from Lambda because it is not the Publicly accessible endpoint, I have created this Private API to be accessible only from the VPC endpoint deployed within the VPC.
4d. Create custom domain name for Private API
1. Choose Custom domain names from the main navigation pane
2. Choose Create
3. For Domain name, enter a domain name
4. Under Configuration, choose Regional
5. Choose a minimum TLS version
6. Choose an ACM certificate. The certificate must be in the same Region as the API
7. Choose Create
8. Follow the Route 53 documentation on configuring Route 53 to route traffic to API Gateway.
The following procedure sets up API mappings to map paths from your custom domain name to your API stages.
1. Choose a custom domain name in the API Gateway
2. Choose Configure API mappings
3. Choose Add new mapping
4. Specify the API, Stage, and Path for the mapping
5. Choose Save
More on Setting up custom domain names for REST APIs
We have our First Private API with two API resources (default and test-resource) and custom domain name — ‘pvt-api.awshero.co.uk’
Repeat step 4 to create a second Private API with Integration with Lambda ‘Hello-lambda3’, attach a Resource policy and associate a custom domain name with API mapping. This API will be treated as Second API with default resource.
I have gone ahead and created and Second API and deployed it -
I have also created the custom domain name ‘pvt-api2.awshero.co.uk’ and configured the API mapping –
Now that I have TWO APIs with custom domain name:
1. pvt-api.awshero.co.uk
2. pvt-api2.awshero.co.uk
Step 5: Create Network Load Balancer or Application Load balancer with SSL certificate and target to the VPC Interface endpoints
1. In the navigation pane, under LOAD BALANCING, choose Load Balancers
2. Choose Create Load Balancer
3. For Network Load Balancer, choose Create
4. For Name, type a name for your load balancer
5. For Scheme, select ‘internal’
6. For Listeners, select TLS (Secure TCP) and port as 443
7. For Availability Zones, select the VPC that you used for deploying your workloads. For each Availability Zone that you used to deploy VPC Interface endpoints, select the Availability Zone and the subnets
8. By default, AWS assigns an IPv4 address to each load balancer node from the subnet for its Availability Zone
9. Choose Next: Security Settings
a. For Certificate type: Choose a certificate from ACM
b. For Certificate name: Select the appropriate certificate, same as you used while creating custom domain name in API gateway
c. For Security policy: Select ELBSecurityPolicy-TLS-1–2–2017–1
10. Choose Next: Configure Routing
11. For Target group, keep the default, New target group
a. For Name, type a name for the new target group
b. Keep Protocol as TLS, Port as 443, and Target type as IP
c. For Health checks, keep the default protocol
12. Choose Next: Register Targets
a. Select the application VPC where the VPC Interface endpoints are deployed
b. For IP: enter the IP address of the endpoint (if you have multiple ENIS then enter all IP addresses) and click Add
13. On the Review page, choose Create
14. Load balancer will take some time to create and to evaluate the Health of the Interface Endpoints. You should see the Healthy status after some time. If Unhealthy, then check the Security group of the Interface Endpoints whether they are configured to allow traffic from VPC CIDR.
Step 6: Create Amazon Route 53 Alias records in the Private hosted zone with traffic routing to NLB
- In the navigation pane, choose Hosted zones
- If you already have a hosted zone for your domain, skip to step 5. If you don’t, perform the applicable procedure to create a hosted zone:
a. To route internet traffic to your resources, such as Amazon S3 buckets or Amazon EC2 instances, see Creating a public hosted zone
b. To route traffic in your VPC, see Creating a private hosted zone
3. Choose Create record
4. Select the Simple routing, click next
5. Click Define Simple record as per below –
Important note: Record name in R53 MUST match the Record name created in the Custom domain name in API Gateway. Refer step 4d
Testing
For testing, I have Simulated the On-Premise environment with a VPC and have a VPC Peering connection with application VPC.
Let’s test the two APIs-
Test: Pvt APIs — Amazon domain name with Endpoints
Test: Pvt APIs — custom domain name
Conclusion
I have showcased a workaround on how the Private APIs (Amazon API Gateway) can be invoked securely using custom domain names with SSL certificates, from On-Premise environment.
You can also use AWS Direct Connect to establish a dedicated private connection from an on-premises network to Amazon VPC and access your private API endpoint over that connection by using public DNS names. You cannot use private DNS names to access your private API from an on-premises network.
I hope that you’ll be benefited from this Blog post on how you can Implement a small workaround to overcome the limitation of using custom domain name with Private APIs in Amazon API Gateway.
Happy Learning!!
Be AWSome!
