The StackPath Developer Portal

Integrate our platform directly into your products and dynamically protect your edge.

Get Started    API Reference

Using Terraform to Create a Multi-Cloud Load Balancer

How to create a multi-cloud load balancer at the edge using the StackPath, AWS, and GCP Terraform plugins.

Terraform is a popular tool among developers for managing infrastructure using declarative configuration files and modules. Using the official StackPath Terraform provider, developers can manage their StackPath Edge Compute resources with Terraform.

While the StackPath Terraform provider only supports Edge Compute resources today, we are planning to expand this to other StackPath resources and products in the future!

In this walk-through, we will use Terraform, Traefik, and StackPath Edge Compute to create a multi-cloud load balancer between Google Cloud Platform (GCP) and Amazon Web Services (AWS). We will create two nginx servers, one in Amazon Web Services and one in Google Cloud Platform, and then create a globally deployed Traefik proxy on StackPath's Edge Compute platform configured with an anycast IP address that routes traffic back to the two nginx servers.

This walk-through configures live resources on the StackPath, AWS, and GCP platforms and may incur small financial charges. The examples in this walk-through use sample private IP addresses, but they will have publicly available addresses when run on live systems. Please be careful before applying these configurations.

Let's get started!

Prerequisites

StackPath Account

To begin, you must have a StackPath account. If you do not have an account yet, please register for a new account in the control portal.

StackPath API Credentials

To configure the StackPath Terraform provider you will need API credentials for authenticating with the StackPath API. Follow the Getting Started Guide to create new API credentials for your StackPath account.

StackPath Stack

StackPath resources are organized by stack, and each StackPath account can have one or more stacks. To use the Terraform provider, you will need to first create a stack for your account and grab the ID of the stack.

Google Cloud Platform Account

To use the full example provided by this walk-through you will need a Google Cloud Platform account. However, you could modify the examples in this walk-through to not rely on GCP.

Google API Credentials

To provision GCP resources with Terraform, you will need API credentials to communicate with the GCP API. Follow this guide to create Google API credentials.

Amazon Web Services Account

To use the full example provided by this walk-through you will need an Amazon Web Services (AWS) account. If you do not have an AWS account and do not wish to create one, you can modify this walk-through to work with other providers or existing web servers.

Amazon API Credentials

To provision AWS resources with Terraform, you will need API credentials to configure the AWS Terraform provider. Follow this guide to create AWS API credentials.

Installing Terraform

Note

This walk-through requires Terraform version >= 0.12.

To install Terraform, download the latest release for your operating system from the Terraform downloads page. Unzip the downloaded file and move the binary to somewhere in your $PATH environment variable to make the binary globally available.

If you use macOS and leverage Homebrew, you can run brew install terraform to install Terraform.

Execute the terraform command to verify the Terraform installation.

$ terraform --version
Terraform v0.12.7

If the Terraform binary is installed correctly, then you should see the version of Terraform you have installed. Now that Terraform is installed installed let's install and configure the StackPath provider plugin!

Installing the StackPath Plugin

The StackPath Terraform provider plugin must be installed before we can define StackPath resources in our Terraform configurations. To start, download the latest binary release from GitHub. Be sure to download the correct release based on your platform (e.g. download the amd64 Linux release for 64-bit Linux systems). If your platform isn't listed in the release then please follow the build instructions in the project's README to compile the program for your operating system and architecture.

$ curl --output ./terraform-provider-stackpath_linux_amd64_v0.3.0.tar.gz \
  -L https://github.com/stackpath/terraform-provider-stackpath/releases/download/v0.3.0/terraform-provider-stackpath_linux_amd64_v0.3.0.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   645    0   645    0     0   2239      0 --:--:-- --:--:-- --:--:--  2239
100 14.2M  100 14.2M    0     0  2490k      0  0:00:05  0:00:05 --:--:-- 3249k

$ tar -zxvf terraform-provider-stackpath_linux_amd64_v0.3.0.tar.gz
x ./terraform-provider-stackpath_v0.3.0

Next, move the release binary into the Terraform installation's plugin directory. In Windows this directory is located in %APPDATA%\terraform.d\plugins\<OS>_<ARCH>, on all other systems this directory is located in ~/.terraform.d/plugins/<OS>_<ARCH>. Refer to Terraform's documentation for more information on installing third party plugins.

Using Linux as an example, move the binary into ~/.terraform.d/plugins/linux_amd64/.

$ mv terraform-provider-stackpath_v0.3.0 ~/.terraform.d/plugins/linux_amd64/

Execute the terraform --version command again to verify that the plugin is installed correctly. You should see the version of the plugin outputted with the Terraform version.

$ terraform --version
Terraform v0.12.7
+ provider.stackpath v0.3.0

Now that Terraform is set up, let's define and configure our resources!

Configuring the StackPath Provider

First, create a new directory for our Terraform project. For this walk-through, we will use the directory name"multi-cloud-load-balancer". The name of the directory does not matter, so feel free to change it as you see fit. For the rest of this walk-through we will assume you're working in your newly created project directory.

$ mkdir ~/multi-cloud-load-balancer
$ cd ~/multi-cloud-load-balancer

Terraform reads configuration files in your working directory that end with the .tf extension. These configuration files are written in Terraform's declarative and human-readable configuration language. Resources and configuration settings for Terraform project can be done in a single or separate configuration files, allowing you to organize your resources however works best for you.

Since we are using the StackPath provider, we need to configure it so that the provider can communicate with the StackPath API. Create a new file in your working directory called provider.tf. This will contain our StackPath provider specific configuration.

variable "stackpath_stack" {}
variable "stackpath_client_id" {}
variable "stackpath_client_secret" {}

provider "stackpath" {
  stack         = "${var.stackpath_stack}"
  client_id     = "${var.stackpath_client_id}"
  client_secret = "${var.stackpath_client_secret}"
}

This configuration defines three variables used when executing Terraform to configure the StackPath provider.

  • stackpath_stack - The ID of the stack that all resources should be created in
  • stackpath_client_id - The client ID for your API credentials
  • stackpath_client_secret - The client secret for your API credentials

Initializing Terraform

Now that we've setup the StackPath provider, we need to initialize Terraform to set up the project. The terraform init command reads through your configuration files and set up any plugins needed for your necessary providers. Note that the command's output specifies the version of the StackPath provider.

$ terraform init
Initializing the backend...

Initializing provider plugins...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.stackpath: version = "~> 0.3"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Now that Terraform is initialized it's time to create resources.

Creating the Backend Web Servers

These steps are optional if you have existing web servers to proxy requests to.

Let's configure the two web servers on CGP and AWS using their Terraform providers.

Creating the AWS Web Server

First, configure the AWS provider in provider.tf as defined in Terraform's AWS provider documentation. Then, copy this sample configuration to an aws.tf file to create a new AWS web server. This Terraform configuration creates an Ubuntu virtual machine running nginx in a new Amazon Virtual Private Cloud. Since we're adding another provider we need to initialize Terraform again using the terraform init command. If all goes well you should see something similar to the following:

$ terraform init
...
* provider.aws: version = "~> 2.19"
...

Execute the terraform apply command to create the web server in AWS. If the command is successful you should see the AWS instance's IP address in the command output.

$ terraform apply \
  -var stackpath_stack=${STACK_ID} \
  -var stackpath_client_id=${CLIENT_ID} \
  -var stackpath_client_secret=${CLIENT_SECRET}

...
aws_vpc.main: Creation complete after 3s
aws_internet_gateway.main: Creation complete after 2s
aws_subnet.main: Creation complete after 2s
aws_security_group.allow_outbound_traffic: Creation complete after 2s
aws_security_group.allow_inbound_http: Creation complete after 3s
aws_default_route_table.main: Creation complete after 1s
aws_instance.web_server_01: Creation complete after 34s

Apply complete! Resources: 7 added, 0 changed, 0 destroyed.

Outputs:

aws-nginx-ip = 10.0.0.1

Once cloud init has run on the server you should be able to reach the nginx server at the IP address provided in your output.

Creating the GCP Web Server

As with the AWS provider, start by configuring the GCP provider in provider.tf as defined in Terraform's GCP provider documentation.

Next, use the provided example gcp.tf Terraform configuration to create a new nginx web server in GCP. After creating the file and configuring the provider, initialize Terraform again with terraform init.

$ terraform init
...
* provider.google: version = "~> 2.10"
...

After the provider is setup, run the terraform apply command again to create the web server in GCP. If all goes well, you should see output similar to the following:

$ terraform apply \
  -var stackpath_stack=${STACK_ID} \
  -var stackpath_client_id=${CLIENT_ID} \
  -var stackpath_client_secret=${CLIENT_SECRET}

...
google_compute_instance.default: Creation complete after 29s [id=web-server-01]
...
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

gcp_instance_ip = 10.0.0.2

Once the start up script runs on the new GCP server you should be able to access the web page using the IP address in the output.

Now let's look at creating a globally distributed proxy for our two web servers using Edge Compute.

Creating the Global Load Balancer at StackPath

StackPath Provider Resources

The StackPath Terraform provider supports the ability to manage Edge Compute workloads and network policies. Define them with the following resource types:

  • stackpath_compute_workload - An Edge Compute workload
  • stackpath_compute_network_policy - An Edge Compute network policy that should apply to one or more workloads.

Creating an Edge Compute Workload

StackPath Edge Compute workloads define a template that should be used to create instances of the workload in locations based on target selectors. An Edge Compute workload can define either containers or virtual machines. For this example we'll define our Edge Compute container workload in a file called traefik-proxy.tf:

resource "stackpath_compute_workload" "traefik-lb" {
  name = "traefik-lb"
  slug = "traefik-lb"

  annotations = {
    # request an anycast IP for a workload
    "anycast.platform.stackpath.net" = "true"
  }

  labels = {
    "role" = "web-server"
  }

  network_interface {
    network = "default"
  }
}

This configuration defines an Edge Compute workload called traefik-lb that has a single network interface for the "default" network. Currently, Edge Compute has the concept of a single network, but once multiple networks are supported you will be able to configure which network the instances should have an interface for. A network in Edge Compute defines a global virtual private cloud (VPC). Instances in the same VPC communicate with each other their private IPs.

The anycast.platform.stackpath.net annotation in workload requests an anycast IP from StackPath. This IP will be announced from all StackPath PoP locations and balance requests across all of the workload's instances in the PoP closest to end users.

This configuration defines a custom label of "role" with the value of "web-server". With labels you can provide arbitrary key/value pairs that can be used in selectors, which expand the capabilities of your Edge Compute workoad.

Creating a Container in the Workload

Now let's start defining the container, add the following configuration to the traefik-lb resource in traefik-proxy.tf:

  container {
    name = "app"
    image = "scotwells/multi-cloud-traefik:v1.0.0"
    resources {
      requests = {
        "cpu"    = "1"
        "memory" = "2Gi"
      }
    }
    env {
      key   = "BACKEND_1"
      value = "http://${aws_instance.web_server_01.public_ip}/"
    }
    env {
      key   = "BACKEND_2"
      value = "http://${google_compute_instance.default.network_interface.0.access_config.0.nat_ip}/"
    }

This defines a container named app that uses the scotwells/multi-cloud-traefik:v1.0.0 image and requests 1 CPU and 2GiB of RAM. The scotwells/multi-cloud-traefik:v1.0.0 image creates a basic Traefik configuration that allows us to define two backends using the environment variables BACKEND_1 and BACKEND_2 and spreads the load evenly across the two. In the above configuration, we pull the public IP of the nginx instance running in AWS to set the BACKEND_1 environment variable and the public IP of the GCP instance as the BACKEND_2 environment variable.

If you chose not to leverage the sample GCP and AWS Terraform configurations, then you'll need to modify this configuration to point to two web servers you define.

Now let's define liveness and readiness and probes for the container to ensure the workload stays up in the event of failure. Add the following configuration to the container section in traefik-proxy.tf:

    liveness_probe {
      period_seconds = 10
      success_threshold = 2
      failure_threshold = 3
      initial_delay_seconds = 30
      http_get {
        port = 80
        path = "/"
        scheme = "HTTP"
      }
    }
    readiness_probe {
      period_seconds = 10
      success_threshold = 2
      failure_threshold = 3
      initial_delay_seconds = 30
      http_get {
        port = 80
        path = "/"
        scheme = "HTTP"
      }
    }

Liveness probes determine the health of an Edge Compute instance. An instance with a failing liveness probe is automatically restarted by StackPath. Readiness probedetermine when an instance is ready to begin serving traffic after the instance is started. Requests made to the anycast IP are only sent to instances with a healthy readiness probe.

  • period_seconds - Defines how often the probe should be executed
  • success_threshold - Defines the number of successful probes needed to mark the instance as "healthy" or "ready"
  • failure_threshold - Defines the number of failed probes needed to mark an instance as "unhealthy" or "not ready"
  • initial_delay_seconds - Defines the number of seconds the system should wait after an instance has started to start probes
  • http_get - Defines the configuration for an HTTP Probe
  • port - Defines the port that should be used in the HTTP request
  • path - Defines the path that should be used in the HTTP request
  • scheme - Defines the scheme that should be used in the HTTP request (HTTP or HTTPS)

Now that our container is fully defined, lets set up our workload targets.

Configuring Workload Target Locations

A target defines where a workload's instances should be created around the globe. Multiple targets can be defined to launch different numbers of instances in specific locations or specify different auto-scaling configurations.

Start by adding the following configuration to the traefik-lb resource in traefik-proxy.tf:

  target {
    name             = "global"
    deployment_scope = "cityCode"
    selector {
      key      = "cityCode"
      operator = "in"
      values = [
        "IAD","JFK","ORD","ATL","MIA",
        "DFW","DEN","SEA","LAX","SJC",
        "YYZ","AMS","LHR","FRA","WAW",
        "SIN","GRU","MEL","NRT","MAD",
        "ARN","HKG",
      ]
    }
  }

This defines a new target called "global" that requests a minimum of 2 replicas in each of the defined cities.

  • name - Uniquely identifies a target within a workload and must be a valid DNS label as described by RFC 1123. Names can only contain the characters 'a-z', '0-9', '-', and '.'
  • deployment_scope - The scope of where an instance should be deployed, the only option currently supported is cityCode. A cityCode is defined by its IATA airport code
  • selector - Defines the location selectors for the workload, multiple can be specified
  • key - Defines the key that should be used to select on, the only option currently supported option is cityCode
  • operator - Defines how the selector should be applied, the only option currently supported is in
  • values - Defines the values that the selector should attempt to match against, here we define all of the current EdgeCompute city locations.

Configuring Auto-Scaling in the Workload's Targets

Now that the workload's targets are defined it's time to configure auto-scaling. Auto-scaling configuration is done at the target level allowing you to create different scaling configurations for each target. Auto-scaling defines the minimum and maximum number of instance replicas that should be created in a selected target. It also allows you to provide metrics and usage thresholds that should be used to determine when to scale instances.

Add the following configuration to the target section in traefik-proxy.tf:

    min_replicas = 2
    max_replicas = 5
    scale_settings {
      metrics {
        metric = "cpu"
        # scale up when CPU averages 50%
        average_utilization = 50
      }
    }

This creates an auto-scaling configuration for the target that ensures a minimum of two instances running per location while never exceed five instances. It increases the number of instances when average CPU utilization is 50% across all instances in the cluster.

  • min_replicas - Defines the minimum number of replicas that should be created in a selected location
  • max_replicas - Defines the maximum number of replicas that should be created in a selected location
  • scale_settings - Defines the configuration for when to scale instances
  • metrics - Defines the thresholds for a metric that should be used to scale a workload instance

Finding the Workload's Anycast IP

Finally, let's configure Terraform to output the anycast IP that was provisioned for the Edge Compute workload. Add the following to traefk-proxy.tf:

output "traefik-anycast-ip" {
  value = replace(
    lookup(
        stackpath_compute_workload.traefik-lb.annotations,
        "anycast.platform.stackpath.net/subnets",
        ""
    ),
    "/32",
    ""
  )
}

This configures an output variable from the workload's subnet allocation for Terraform to display.

Applying the Configuration

Now let's apply the newly created Terraform configuration to start our global load balancer. Run the terraform apply command and look for the load balancer's anycast IP in the output.

$ terraform apply \
  -var stackpath_stack=${STACK_ID} \
  -var stackpath_client_id=${CLIENT_ID} \
  -var stackpath_client_secret=${CLIENT_SECRET}

...
stackpath_compute_workload.traefik-lb: Creating...
stackpath_compute_workload.traefik-lb: Creation complete after 1s

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:
...
traefik-anycast-ip = 10.0.0.3

The workload's instances will start up in the locations defined in our target selector within a few seconds of being created. Once they'reready the anycast IP will route traffic to each load balancer instance.

Outputting Workload Instances

Terraform can be configured to output the state of the individual instances running in your Edge Compute workload. Add the following to traefik-proxy.tf:

output "traefik-workload-instances" {
  value = {
    for instance in stackpath_compute_workload.traefik-lb.instances :
    instance.name => {
      "ip_address" = instance.external_ip_address
      "phase"      = instance.phase
    }
  }
}

This configures Terraform to output the current phase of the workload instance along with it's provisioned public IP address. Run the terraform refresh command to apply these changes and see the new output.

$ terraform refresh \
  -var stackpath_stack=${STACK_ID} \
  -var stackpath_client_id=${CLIENT_ID} \
  -var stackpath_client_secret=${CLIENT_SECRET}

...
stackpath_compute_workload.traefik-lb: Refreshing state...
...

Outputs:

...
traefik-workload-instances = {
  "traefik-lb-us-jfk-0" = {
    "ip_address" = "10.0.0.5"
    "phase" = "RUNNING"
  }
  "traefik-lb-us-sea-0" = {
    "ip_address" = "10.0.0.6"
    "phase" = "RUNNING"
  }
}

Instances are available via its public IP and the workload's anycast IP after they reach the RUNNING phase. However, if you open your web browser and navigate to one of the IPs, you'll notice that the connection times out. What's going on?

By default, StackPath blocks all public traffic to an instance. To be able to access the instance on port 80, we will need to create an Edge Compute network policy to allow traffic to instances on port 80.

Creating a Global Network Policy

StackPath network policies leverage selectors to determine the networks and workload instances that a network policy should apply to. A network policy created with no selectors will automatically apply to all networks and all instances created in a stack. For this example we are going to create a global network policy that will allow anyone to access port 80 of a workload instance with the role of "web-server".

Create a new Terraform configuration file called web-server-network-policy.tf with the following contents:

resource "stackpath_compute_network_policy" "web-server" {
  name        = "Allow HTTP traffic for web servers"
  slug        = "web-servers-allow-http"
  description = "A network policy for allowing HTTP access for instances with the web server role"
  priority    = 20000

  instance_selector {
    key      = "role"
    operator = "in"
    values   = ["web-server"]
  }
}

This defines a new stackpath_compute_network_policy Terraform resource named web-server with a priority level of 20,000. Priority levels are unique among network policies on a stack and must be a number between 1 and 65,000. Higher priority network policies override lower priority policies.

Notice how the instance_selector here uses the same role label defined in traefik-proxy.tf. This applies the network policy to our global load balancer workload.

Now that we have the general configuration of the network policy set up, let's configure an ingress rule to allow port 80 access from all internet traffic.

Add the following configuration to web-server-network-policy.tf:

  policy_types = ["INGRESS"]
  ingress {
    action      = "ALLOW"
    description = "Allow port 80 traffic from all IPs"
    protocol {
      tcp {
        destination_ports = [80]
      }
    }
    from {
      ip_block {
        cidr = "0.0.0.0/0"
      }
    }
  }

This creates a new ingress policy that allows traffic from 0.0.0.0/0 (all IPv4 IPs) to access all web-server instances running in the stack on port 80.

  • policy_types - Allows us to specify what type of network policy we want to create and whether that policy will apply to ingress or egress traffic
  • ingress - Defines the ingress rules for the policy, multiple ingress blocks can be provided
  • action - Configures how the policy should treat traffic based on the ingress configuration,either "ALLOW" or "BLOCK"
  • protocol - The ingress protocol's configuration. In this case saying that the policy should apply to any TCP traffic destined for port 80
  • from - The source of traffic that the ingress policy should apply to. By default the policy applies to all traffic, but here we specify that the ingress policy should apply to all IPv4 IPs by using the CIDR block 0.0.0.0/0.

For more networking options, reference the full network policy example on GitHub.

Applying the Network Policy With Terraform

Now that we have our network policy defined, lets use Terraform to create it!

Run the terraform plan command to see what Terraform needs to do to build the infrastructure. Be sure to pass in the variables we defined earlier so our provider can communicate with the StackPath API. If everything is configured correctly, you should see output similar to the following:

$ terraform plan \
  -var stackpath_stack=${STACK_ID} \
  -var stackpath_client_id=${CLIENT_ID} \
  -var stackpath_client_secret=${CLIENT_SECRET}

Refreshing Terraform state prior to plan...
...
Terraform will perform the following actions:

  # stackpath_compute_network_policy.web-server will be created
...

This output shows that Terraform will create a new network policy called "web-server" along with all the options that will be used. Finally, create the network policy with the terraform apply command. If all goes well, you should be able to get output similar to the following:

$ terraform apply \
  -var stackpath_stack=${STACK_ID} \
  -var stackpath_client_id=${CLIENT_ID} \
  -var stackpath_client_secret=${CLIENT_SECRET}

...
stackpath_compute_network_policy.web-server: Creating...
stackpath_compute_network_policy.web-server: Creation complete after 1s
...
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Wrapping Up

With the Terraform StackPath provider, we can configure StackPath Edge Compute resources using Terraform configurations. We've been able to setup an entire infrastructure in AWS and GCP and load balance across the two cloud platforms using a global anycast Edge Compute workload with Traefik.

We could improve this walk-through by creating web servers in several regions around the globe and configure EdgeCompute to proxy traffic to the closest region based on the location the server launched in, but we'll leave that for another article.

Using Terraform to Create a Multi-Cloud Load Balancer


How to create a multi-cloud load balancer at the edge using the StackPath, AWS, and GCP Terraform plugins.

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.