Connect On-Prem Services Using AWS ALB

Owned by Brandon Godwin
Feb 19, 2021
5 min read

Prerequisites

  • You must have at least two public subnets in your VPC for use by the ALB. These subnets should have a route table with the default route pointing to an Internet Gateway that’s attached to the VPC.

    • These subnets should be in different Availability Zones.

    • They should have a CIDR block of at least /27.

    • They should each have at least 8 free IP addresses available.

  • You will need a TLS certificate registered in AWS Certificate Manager for use with the HTTPS listener of the ALB.

    • The TLS certificate should have Subject Alternative Names (SAN) that cover the hostnames of each service that will be connected to Allstacks. You can utilize a wildcard SAN for this, or specify them individually. For example, the SAN *.example.com will cover jira.example.com and gitlab.example.com.

    • AWS Certificate Manager supports issuing certificates from the widely trusted Amazon public Certificate Authority, or importing existing TLS certificates that have been signed by third party Certificate Authorities. If you allow ACM to issue the certificate, it will automatically handle renewal of the certificate before it expires. For more information on ACM issued certificates, see this AWS documentation.

  • If you do not intend to utilize the ALB for end user traffic as well, you will need to be able to support a split-horizon DNS configuration. This is required so that Allstacks can access the service using the same hostname that internal users use and that the service is configured to expect. This also means using HTTPS for internal traffic to the service and configuring the service appropriately so that it generates links with the HTTPS scheme and hostname that Allstacks will be using to access it.

It is possible to work around this split-horizon DNS requirement in some circumstances. Please contact Allstacks support for more information.

AWS ALB Setup and Configuration

  1. Create a new security group for use by the ALB. This security group should allow inbound traffic on port 80 and 443.

    1. If you do not intend to use this ALB for end user access, you can whitelist the specific IP addresses that Allstacks will use to make API requests. Please contact Allstacks support for the list of IP addresses to whitelist.

  2. Update the security group for the service that will be exposed through this load balancer to allow traffic from the newly created security group used by the ALB.

  3. In the AWS console, navigate to the Load Balancers section of the EC2 console and create a new Application Load Balancer.

    1. Choose a descriptive name for the new ALB. A single ALB can be used for multiple integrations by Allstacks, so you can use something like allstacks-integration.

    2. Add an HTTPS listener on port 443.

    3. Chose the VPC with your services, and check at least two of the availability zones that you have public subnets in. Chose these public subnets from the dropdowns. It’s recommended that at least one of these availability zones matches up with the AZ that your service is running in.

  4. Continue to the next step to configure TLS settings.

    1. Chose the ACM certificate that will cover the hostnames of the services being connected through this ALB.

    2. It’s recommended that you set a security policy that disables older TLS protocol versions, such as ELBSecurityPolicy-TLS-1-2-Ext-2018-06.

  5. Continue to the next step to configure security groups for the ALB. Select the security group that you created in step 1.

  6. Continue to the next step to configure the first target group, which determines where the ALB forwards traffic. You can create more target groups later to connect more services.

    1. Configure the protocol and port for the target group to match how the internal service accepts traffic.

    2. For the health check path, specify a path that will return a HTTP status code of 200 when the service is functioning properly. For instance, this could be a login page, such as /users/sign_in for Gitlab. If the health check path is not correct, you will see an error like below on the target group details page after you’ve finished creating the load balancer. This won’t prevent the ALB from functioning properly, but makes it more difficult to monitor and set up alerting for health issues related to this integration.

  7. Continue to the next step to configure the EC2 instance(s) that the target group will point to. Select the instance from the list and click the Add to registered button.

  8. Continue through the wizard to finish creating the load balancer. It will take a few minutes to provision. Once that’s finished, the state will change from provisioning to active on the load balancers list page of the EC2 console.

  9. The the new load balancer is in the active state, click the Listeners tab and click View/edit rules for the HTTP listener. Click the pencil icon to edit existing rules, then delete the Forward action and replace it with a Redirect action to redirect to HTTPS. The config should look like below. Save the config and return to the Listeners tab of the load balancer details page.

  10. Click the View/edit rules link for the HTTPS listener. Modify the existing default rule to return a fixed response of 404. Then create a new rule to forward to the target group you created earlier when the Host header matches the hostname being used for the service. The config should look like below. You can use this page to continue adding more services for this ALB to forward traffic to. You’ll need to create the target groups for each service first, then they will be available for use here as long as another load balancer isn’t using the target group.

  11. On the Description tab for the ALB, there’s a DNS name property for the load balancer. It’s important to use this as the public IPs for an ALB can change over time. Create new or update the existing DNS records for the services you intend to expose through this ALB. These records should be CNAME records that point to the DNS name value of the load balancer.

Monitoring the Integration

The ALB exposed various CloudWatch metrics that can be helpful for monitoring the health of the service integration with Allstacks. Alarms can be configured on these CloudWatch metrics to send alerts via email, for example.

  • Target Response Time: A high target response time can indicate that the service is under heavy load.

  • HTTP 5xx: A large number of 5xx errors from the service can indicate internal errors in the service itself, such as bad configuration or inability to connect with an external dependency such as a database.

  • HTTP 4xx: A large number of 4xx errors from the service can indicate an authentication or authorization problem with the user that has connected the service in Allstacks.

  • ELB 5xx: ELB 5xx errors could indicate that the ALB is unable to communicate with the service, which can be caused by bad firewall or security group configuration. It can also indicate that the response timeout of the ALB needs to be increased. By default, the ALB will return a gateway timeout if the backend service takes longer than 60 seconds to respond to the request.