The StackPath Developer Portal

Integrate our platform directly into your products and dynamically protect your edge.

Get Started    API Reference

Using Terraform to create a Multi-Cloud Load Balancer

How to create a multi-cloud load balancer at the Edge using the StackPath, AWS, and GCP terraform plugins.

Introduction

Terraform is a popular tool among developers for managing infrastructure using declarative configuration files and modules.
Using the official StackPath Terraform provider, developers can manage their StackPath EdgeCompute resources with Terraform.

While the StackPath Terraform provider only supports EdgeCompute resources today, we are planning to expand this to other StackPath resources and products in the future!

In this walk-through, we will use Terraform, Traefik, and StackPath EdgeCompute to create a multi-cloud load balancer between Google Cloud Platform and Amazon Web Services.
We will create two nginx servers, one in Amazon Web Services and one in Google Cloud Platform, and then create a globally deployed Traefik proxy configured with an anycast IP address that routes traffic back to our two nginx servers.

Let's get started!

Prerequisites

StackPath Account

This walkthrough will require you to have an existing StackPath account.
If you do not have an account, you can register for a new account in the control portal.

StackPath API Credentials

To configure the StackPath Terraform provider you will need API Credentials for authenticating with the StackPath API.
To create new API Credentials for you StackPath account, follow the Getting Started Guide.

StackPath Stack

StackPath resources are created within a Stack and each StackPath account can have one or more Stacks.
To use the Terraform provider, you will need to first create a Stack for your account and grab the ID of the Stack.

Google Cloud Platform Account

To use the full example provided by this walk-through you will need a Google Cloud Platform (GCP) account.
However, you could modify the examples in this walk-through to not rely on GCP.

Google API Credentials

To provision GCP resources with Terraform, you will need API credentials to communicate with the GCP API.
Follow this guide to create Google API Credentials.

Amazon Web Services Account

To use the full example provided by this walk-through you will need an Amazon Web Services (AWS) account.
If you don't have an AWS account and do not wish to create one, you can modify this walk-through to work with other providers or existing web servers.

Amazon API Credentials

To provision AWS resources with Terraform, you will need API credentials to configure the AWS Terraform provider.
Follow this guide to create AWS API Credentials.

Installing Terraform

Note: This walk-through uses some of the newer features released in Terraform 0.12

To install Terraform, download the latest release for your operating system from the Terraform Downloads page.
Unzip the downloaded file and move the binary to somewhere in your PATH environment variable to make the binary globally available.

If you use macOS and leverage homebrew, you can run brew install terraform to install Terraform.

To verify the Terraform installation, you can execute the terraform command.

$ terraform --version
Terraform v0.12.3

If you installed the Terraform binary correctly, you should see the version of Terraform you have installed.
Now that we have Terraform installed, let's install and configure the StackPath provider plugin!

Installing the StackPath Plugin

Before you can define StackPath resources in your Terraform configurations, you will need to manually install the StackPath provider as a Terraform plugin.
To install the StackPath provider, download the latest binary release from GitHub.
Be sure to download the correct release based on your platform (eg for linux download the linux release.)

wget -q https://github.com/stackpath/terraform-provider-stackpath/releases/download/v0.3.0/terraform-provider-stackpath_linux_amd64_v0.3.0.zip -o /dev/null -O terraform-provider-stackpath_linux_amd64_v0.3.0.zip
unzip terraform-provider-stackpath_linux_amd64_v0.3.0

After you've downloaded and unzipped the release, you will need to move the release binary into the plugin directory for your Terraform installation.
On windows this directory is located in %APPDATA%\terraform.d\plugins\<OS>_<ARCH>, on all other systems this directory is located in ~/.terraform.d/plugins/<OS>_<ARCH>.
Refer to the Terraform documentation for more information on installing third party plugins.

Using linux as an example, you would move the binary into ~/.terraform.d/plugins/linux_amd64/.

mv terraform-provider-stackpath_v0.3.0 ~/.terraform.d/plugins/linux_amd64/

To verify that the plugin is installed correctly, execute the terraform --version command again.

$ terraform --version
Terraform v0.12.3
+ provider.stackpath v0.3.0

If the plugin is installed correctly, you should see the version of the plugin outputted with the Terraform version.
Now that we're all set up with Terraform, lets begin defining and configuring resources!

Configuring the StackPath Provider

First we need to create a new directory for our Terraform project.
For this walkthrough, we are going with the name "multi-cloud-load-balancer" for our directory.
The name of the directory does not matter, so feel free to change it as you see fit.

mkdir ~/multi-cloud-load-balancer
cd ~/multi-cloud-load-balancer

For the rest of this walkthrough we will assume you're working in your newly created project directory.

Terraform works by reading configuration files in your working directory that end in the .tf extension.
Terraform provides its own configuration language that is used to create these configuration files and promotes a declarative and human readable format.
Resources and configuration settings for Terraform can be done in a single configuration file or separate configuration files, allowing you to organize your resources however works best for you.

Since we will be using the StackPath provider, we need to configure it so that the provider can communicate with the StackPath API.

Create a new file in your working directory called provider.tf which will contain all of our StackPath provider specific configuration.

vi provider.tf

Add this provider configuration to the provider.tf file.

variable "stackpath_stack" {}
variable "stackpath_client_id" {}
variable "stackpath_client_secret" {}

provider "stackpath" {
    stack         = "${var.stackpath_stack}"
    client_id     = "${var.stackpath_client_id}"
    client_secret = "${var.stackpath_client_secret}"
}

This configuration defines three variables we can set when executing Terraform to configure the StackPath provider.

  • stackpath_stack - The ID of the stack that all resources should be created in
  • stackpath_client_id - The client ID for your API credentials
  • stackpath_client_secret - The client secret for your API credentials

The provider is then configured with the values from the defined variables.
These are the only required settings when configuring the StackPath provider.

Initializing Terraform

Now that we've setup the StackPath provider, we need to initialize Terraform to setup the project.
Initializing Terraform will read through your configuration files and set up any plugins needed for your necessary providers.

terraform init

If your project is setup correctly, you should see output similar to the following.
Notice how to specifies the version of the StackPath provider you currently have installed.

Initializing the backend...

Initializing provider plugins...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.stackpath: version = "~> 0.2"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

StackPath Provider Resources

The StackPath Terraform provider currently supports the ability to manage EdgeCompute workloads and network policies.
You can define them using the following resource types:

  • stackpath_compute_workload - An EdgeCompute workload
  • stackpath_compute_network_policy - An EdgeCompute network policy that should apply to one or more workloads.

Let's start by creating a Traefik workload setup with an anycast IP that will load balance requests between an AWS web server and a GCP web server.

Creating a Multi-Cloud Load Balancer

First let's configure two web servers in different providers.
For this example, we will configure two web servers running nginx; one running in AWS and one running in GCP.

Creating an AWS Web Server

This step is optional if you have an existing web server you can use to proxy requests to.

Make sure you configure the AWS provider in provider.tf.
See the official AWS provider documentation for help configuring the provider.

To create a new AWS web server, you can use this example Terraform configuration.
This Terraform configuration will create a new Ubuntu VM running nginx in a new VPC.
Copy the contents of aws.tf in the above example to a new file called aws.tf.
Now since we're adding another provider into the mix we need to initialize Terraform again using the terraform init command.

If all goes well you should see something similar to the following:

$ terraform init
...
* provider.aws: version = "~> 2.19"
...

Now that the AWS provider has been installed and setup, you can execute the terraform apply command to create the web server in AWS.

terraform apply \
  -var stackpath_stack=${STACK_ID} \
  -var stackpath_client_id=${CLIENT_ID} \
  -var stackpath_client_secret=${CLIENT_SECRET}

If the command is successful, the output should be similar to the following:

...
aws_vpc.main: Creation complete after 3s
aws_internet_gateway.main: Creation complete after 2s
aws_subnet.main: Creation complete after 2s
aws_security_group.allow_outbound_traffic: Creation complete after 2s
aws_security_group.allow_inbound_http: Creation complete after 3s
aws_default_route_table.main: Creation complete after 1s
aws_instance.web_server_01: Creation complete after 34s

Apply complete! Resources: 7 added, 0 changed, 0 destroyed.

Outputs:

aws-nginx-ip = 3.86.234.111

Once cloud init has run on the server you should be able to reach the nginx server at the IP address provided in your output.

Creating a GCP Web Server

Make sure you configure the GCP provider in provider.tf. See the official GCP provider documentation for help configuring the provider.

Just like with the AWS web server, you can use the provided example gcp.tf Terraform configuration to create a new nginx web server in GCP.
After creating the file and configuring the provider, initialize Terraform again with terraform init.

If your configuration is setup correctly you should see output similar to the following:

$ terraform init
...
* provider.google: version = "~> 2.10"
...

Once the provider is setup, use the terraform apply command again to create the web server in GCP.
If all goes well, you should see output similar to the following:

...
google_compute_instance.default: Creation complete after 29s [id=web-server-01]
...
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

gcp_instance_ip = 35.238.217.208

Once the start up script runs on the new GCP server, you should be able to access the web page using the IP address in the output.
Now let's look at creating a globally distributed proxy for our two web servers using EdgeCompute.

A full example can be found at (https://github.com/stackpath/terraform-provider-stackpath/blob/master/examples/multi-cloud-load-balancer/traefik-proxy.tf)

Creating an EdgeCompute Workload

StackPath EdgeCompute workloads define a template that should be used to create instances of the workload in locations based on the target selectors.
An EdgeCompute workload can define either Containers or Virtual Machines, for this example we will configure a container based workload.

Start by creating a new Terraform configuration file called traefik-proxy.tf.

vi traefik-proxy.tf

Create the following EdgeCompute workload:

resource "stackpath_compute_workload" "traefik-lb" {
  name = "traefik-lb"
  slug = "traefik-lb"

  annotations = {
    # request an anycast IP for a workload
    "anycast.platform.stackpath.net" = "true"
  }

  labels = {
    "role" = "web-server"
  }

  network_interface {
    network = "default"
  }

This configuration will define an EdgeCompute workload called traefik-lb that has a single network interface for the "default" network.
Currently EdgeCompute only has the concept of a single network, but once multiple networks are supported you will be able to configure which network the instances should have an interface for.
A network in EdgeCompute defines a global VPC, instances in the same VPC will be able to communicate over their private IPs.

The anycast.platform.stackpath.net annotation added to the workload will request an anycast IP for the workload.
The anycast IP will be announced from all PoP locations and will balance requests across all of the instances for the workload in the PoP closest to end-users.

We're also adding a custom label of "role" with the value of "web-server".
With labels you can provide arbitrary key/value pairs that can be used in selectors, more on what we can do with that later.

Creating a Container

Now let's start defining the container, add the following Terraform configuration:

  container {
    name = "app"
    image = "scotwells/multi-cloud-traefik:v1.0.0"
    resources {
      requests = {
        "cpu"    = "1"
        "memory" = "2Gi"
      }
    }
    env {
      key   = "BACKEND_1"
      value = "http://${aws_instance.web_server_01.public_ip}/"
    }
    env {
      key   = "BACKEND_2"
      value = "http://${google_compute_instance.default.network_interface.0.access_config.0.nat_ip}/"
    }

This will define a container named app that uses the scotwells/multi-cloud-traefik:v1.0.0 image and requests 1 CPU and 2GiB of RAM.
The scotwells/multi-cloud-traefik:v1.0.0 image creates a basic Traefik configuration that allows us to define 2 backends using environment variables BACKEND_1 and BACKEND_2 and spreads the load evenly across the two.
In the above configuration, we pull the public IP of the nginx instance running in AWS to set the BACKEND_1 environment variable and the public IP of the GCP instance as the BACKEND_2 environment variable.

If you chose not to leverage the GCP and AWS Terraform configurations earlier, you'll need to modify this configuration to point to two web servers you define.

Now let's define readiness and liveness probes for the container.
Add the following configuration to your Terraform file:

    liveness_probe {
      period_seconds = 10
      success_threshold = 2
      failure_threshold = 3
      initial_delay_seconds = 30
      http_get {
        port = 80
        path = "/"
        scheme = "HTTP"
      }
    }
    readiness_probe {
      period_seconds = 10
      success_threshold = 2
      failure_threshold = 3
      initial_delay_seconds = 30
      http_get {
        port = 80
        path = "/"
        scheme = "HTTP"
      }
    }
  }

The liveness probe is used to determine the health of an instance, an instance with a failing liveness probe will be restarted by the system.
The readiness probe is used to determine when an instance is ready to begin serving traffic, after the instance is started, requests made to the anycast IP will only be sent to instances with a healthy readiness probe.

  • period_seconds - Defines how often the probe should be executed
  • success_threshold - Defines the number of successful probes needed to mark the instance as "healthy" or "ready"
  • failure_threshold - Defines the number of failed probes needed to mark an instance as "unhealthy" or "not ready"
  • initial_delay_seconds - Defines the number of seconds the system should wait after an instance has started to start probes
  • http_get - Defines the configuration for an HTTP Probe
  • port - Defines the port that should be used in the HTTP request
  • path - Defines the path that should be used in the HTTP request
  • scheme - Defines the scheme that should be used in the HTTP request (HTTP or HTTPS)

Now that our container is fully defined, lets set up our workload targets.

Configuring Workload Targets

A target defines where workloads should be created around the globe.
Multiple targets can be defined to launch different numbers of instances in specific locations or specify different auto-scaling configurations.
We currently only support selecting locations based on a CityCode but will eventually expand to other localities.

Start by adding the following configuration to your Terraform configuration:

  target {
    name             = "global"
    deployment_scope = "cityCode"
    selector {
      key      = "cityCode"
      operator = "in"
      values = [
        "IAD","JFK","ORD","ATL","MIA",
        "DFW","DEN","SEA","LAX","SJC",
        "YYZ","AMS","LHR","FRA","WAW",
        "SIN","GRU","MEL","NRT","MAD",
        "ARN","HKG",
      ]
    }

This will define a new target called "global" that requests a minimum of 2 replicas in each selected location.

  • name - Uniquely identifies a target within a workload and must be a valid DNS label as described by RFC 1123, can only container 'a-z', '0-9', '-', and '.'
  • deployment_scope - The scope of where an instance should be deployed, the only option currently supported is cityCode
  • selector - Defines the location selectors for the workload, multiple can be specified
  • key - Defines the key that should be used to select on, the only option currently supported option is cityCode
  • operator - Defines how the selector should be applied, the only option currently supported is in
  • values - Defines the values that the selector should attempt to match against, here we define all of the current EdgeCompute city locations.

With the basis of our target setup to select all EdgeCompute locations, all that's left is configuring auto-scaling.
Auto-scaling configuration is done at the target level allowing you to create different scaling configurations for each target.
Auto-scaling allows you to specify the minimum and maximum number of instance replicas that should be created in a selected target.
It also allows you to provide metrics and usage thresholds that should be used to determine when to scale instances.

Add the following configuration to your Terraform workload configuration:

    min_replicas = 2
    max_replicas = 5
    scale_settings {
      metrics {
        metric = "cpu"
        # scale up when CPU averages 50%
        average_utilization = 50
      }
    }
  }
}

This will create an auto-scaling configuration for the target that will ensure we have a minimum of 2 instances running per location and never exceed 10 instances.
It will also scale up the number of instances when average CPU utilization is 50% across all instances in the cluster.

  • min_replicas - Defines the minimum number of replicas that should be created in a selected location
  • max_replicas - Defines the maximum number of replicas that should be created in a selected location
  • scale_settings - Defines the configuration for when to scale instances
  • metrics - Defines the thresholds for a metric that should be used to scale a workload instance

Lastly, lets configure Terraform to output the anycast IP that was provisioned for the EdgeCompute workload.

Add the following to your Terraform configuration:

output "traefik-anycast-ip" {
  value = replace(
    lookup(
        stackpath_compute_workload.traefik-lb.annotations,
        "anycast.platform.stackpath.net/subnets",
        ""
    ),
    "/32",
    ""
  )
}

This will configure an output variable so that terraform outputs the anycast subnet allocation.

Now let's apply the newly created Terraform configuration, run the terraform apply command:

terraform apply \
  -var stackpath_stack=${STACK_ID} \
  -var stackpath_client_id=${CLIENT_ID} \
  -var stackpath_client_secret=${CLIENT_SECRET}

If everything is configured correctly, you should see output similar to the following:

...
Terraform will perform the following actions:
...
stackpath_compute_workload.traefik-lb: Creating...
stackpath_compute_workload.traefik-lb: Creation complete after 1s

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:
...
traefik-anycast-ip = 185.85.196.5

The workload instances should start spinning up in the locations defined in our target selector within a few seconds of being created.
Once they become ready the anycast IP should begin routing traffic to each instance.

Outputting Workload Instances

If you want to output the status of each of the workload instances running in your workload, you can update your Terraform configuration to have the following output definition.

Note: This will only work in Terraform 0.12+

output "traefik-workload-instances" {
  value = {
    for instance in stackpath_compute_workload.traefik-lb.instances :
    instance.name => {
      "ip_address" = instance.external_ip_address
      "phase"      = instance.phase
    }
  }
}

This will output the current phase of the workload instance along with the public IP provisioned for the instance.
Now when you run the terraform refresh command you should see the IP and phase of each workload instance.

$ terraform refresh \
  -var stackpath_stack=${STACK_ID} \
  -var stackpath_client_id=${CLIENT_ID} \
  -var stackpath_client_secret=${CLIENT_SECRET}

...
stackpath_compute_workload.traefik-lb: Refreshing state...
...

Outputs:

...
traefik-workload-instances = {
  "traefik-lb-us-jfk-0" = {
    "ip_address" = "151.139.31.4"
    "phase" = "RUNNING"
  }
  "traefik-lb-us-sea-0" = {
    "ip_address" = "151.139.124.5"
    "phase" = "RUNNING"
  }
}

Once you see instances are in the RUNNING phase each instance will be available via its public IP and the anycast IP provisioned for your workload.
However, if you open your web browser and navigate to one of the IPs, you'll notice that the connection times out.
What's going on?

By default, all public traffic to an instance is blocked.
To be able to access the instance on port 80, we will need to create an EdgeCompute Network Policy to allow traffic to instances on port 80.

Creating a Global Network Policy

StackPath network policies leverage selectors to determine the networks and workload instances that a network policy should apply to.
A network policy created with no selectors will automatically apply to all networks and all instances created in a stack.
For this example, we are going to create a global network policy that will allow anyone to access port 80 of a workload instance with the role of "web-server".

Create a new Terraform configuration file called web-server-network-policy.tf.

vi web-server-network-policy.tf

Create the following network policy:

resource "stackpath_compute_network_policy" "web-server" {
  name        = "Allow HTTP traffic for web servers"
  slug        = "web-servers-allow-http"
  description = "A network policy for allowing HTTP access for instances with the web server role"
  priority    = 20000

  instance_selector {
    key      = "role"
    operator = "in"
    values   = ["web-server"]
  }

Here we create a new stackpath_compute_network_policy named web-server in Terraform with a priority level of 20,000.
Priority levels are unique among network policies on a Stack and must be a number between 1 and 65,000 inclusive.
Higher priority network policies will overrule any conflicting rules in lower priority policies.

Notice how the instance_selector here is using the same

Now that we have the general configuration of the network policy setup, let's configure an ingress rule to allow port 80 access from all internet traffic.

Add the following configuration to the network policy file.

  policy_types = ["INGRESS"]
  ingress {
    action      = "ALLOW"
    description = "Allow port 80 traffic from all IPs"
    protocol {
      tcp {
        destination_ports = [80]
      }
    }
    from {
      ip_block {
        cidr = "0.0.0.0/0"
      }
    }
  }
}

This creates a new ingress policy that allows traffic from 0.0.0.0/0 (all IPv4 IPs) to access all web-server instances running in the stack on port 80.

  • policy_types - Option allows us to specify what type of network policy we want to create and whether that policy will apply to ingress or egress traffic
  • ingress - Allows us to define the ingress rules for the policy, multiple ingress blocks can be provided
  • action - Configures how the policy should treat traffic based on the ingress configuration, the available actions being "ALLOW" and "BLOCK"
  • protocol - Allows you to specify the ingress protocol configuration, here we're saying that the policy should apply to any TCP traffic destined for port 80
  • from - Allows us to specify the source of traffic that the ingress policy should apply to, by default the policy will apply to all traffic, here we are specifying that the ingress policy should apply to all IPv4 IPs by using the CIDR block 0.0.0.0/0.

For complete options, reference the full network policy example in the GitHub repo.

Apply the Network Policy With Terraform

Now that we have our network policy defined, lets use Terraform to create it!

We can run the terraform plan command to see what Terraform needs to do to build the infrastructure.
Be sure to pass in the variables we defined earlier so that our provider can communicate with the API.

terraform plan \
  -var stackpath_stack=${STACK_ID} \
  -var stackpath_client_id=${CLIENT_ID} \
  -var stackpath_client_secret=${CLIENT_SECRET}

If everything is configured correctly, you should see output similar to the following:

Refreshing Terraform state prior to plan...
...
Terraform will perform the following actions:

  # stackpath_compute_network_policy.web-server will be created
...

This output shows that Terraform will create a new network policy called "web-server" along with all the options that will be used. Now we can create the network policy using the terraform apply command.

terraform apply \
  -var stackpath_stack=${STACK_ID} \
  -var stackpath_client_id=${CLIENT_ID} \
  -var stackpath_client_secret=${CLIENT_SECRET}

If all goes well, you should be able to get output similar to the following:

...
stackpath_compute_network_policy.web-server: Creating...
stackpath_compute_network_policy.web-server: Creation complete after 1s
...
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Now let's create a web server workload to test out our new network policy!

Wrapping Up

With the Terraform StackPath provider, we can now fully configure StackPath EdgeCompute resources using Terraform configurations.
We've been able to setup an entire infrastructure in AWS and GCP and load balance across the two cloud platforms using a global anycast EdgeCompute workload with Traefik.

We could improve this walk-through by creating web servers in several regions around the globe and configure EdgeCompute to proxy traffic to the closest region based on the location the server launched in, but we'll leave that for another article.

Using Terraform to create a Multi-Cloud Load Balancer


How to create a multi-cloud load balancer at the Edge using the StackPath, AWS, and GCP terraform plugins.

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.