Installing and Configuring Consul Cluster in a Amazon VPC (Same subnet):

This series of posts will examine different aspects of running a Consul cluster on different Amazon configurations.

In this post we will look briefly at two things

  • The initial setup in an AWS VPC with all Consul nodes in a single subnet. All nodes were running in an ap-northeast zone.
  • Results of some network tests and conclusion.

Installation

AWS VPC Layout:
VPC Layout

Image Courtesy – https://aws.amazon.com/articles/6079781443936876

Ports used:

Consul requires up to 5 different ports to work properly, some on TCP, UDP and some on both protocols.

  • Server RPC (Default 8300). This is used by servers to handle incoming requests from other agents. TCP only.
  • Serf LAN (Default 8301). This is used to handle gossip in the LAN. Required by all agents. TCP and UDP.
  • Serf WAN (Default 8302). This is used by servers to gossip over the WAN to other servers. TCP and UDP.
  • CLI RPC (Default 8400). This is used by all agents to handle RPC from the CLI. TCP only.
  • HTTP API (Default 8500). This is used by clients to talk to the HTTP API. TCP only.
  • DNS Interface (Default 8600). Used to resolve DNS queries. TCP and UDP.

Consul will also make an outgoing connection to HashiCorp’s servers for Atlas-related features and to check for the availability of newer versions of Consul. This will be a TLS-secured TCP connection to scada.hashicorp.com:7223

AWS Security Group Configuration for VPC:

Consule VPC Security Groups

Clients & Servers:

10.0.20.100 Bootstrap Server
10.0.20.103 Server
10.0.20.104 Server
10.0.20.101 Client

 

We start the cluster in a bootstrap server mode for setting up the cluster initially, however, this is deprecated and the bootstrap-expect flag should be used instead. This does not affect the outcome of our tests.

Consul Installation:

  1. wget https://releases.hashicorp.com/consul/0.6.4/consul_0.6.4_linux_amd64.zip
  2. unzip consul_0.6.4_linux_amd64.zip
  3. chmod +x consul
  4. mv consul /usr/local/bin

Login to Bootstrap Server

  1. Repeat all the steps of Consul Installation.
  2. adduser consul
  3. mkdir -p /etc/consul.d/{bootstrap,server}
  4. mkdir /var/consul
  5. chown consul:consul /var/consul
  6. consul keygen
    1. Output: “Cd9sOxpE5XD9RfHOt3YmEQ== “ (will differ for you)
  7. create a file “ /etc/consul.d/bootstrap/config.json ” and append the following data in it:

 

8. create a file “ /etc/consul.d/server/config.json “ and append the following data in it.

 

Login to Regular (non-bootstrap) Servers

  1. Repeat all the steps of Consul Installation.
  2. adduser consul
  3. mkdir -p /etc/consul.d/server
  4. mkdir /var/consul
  5. chown consul:consul /var/consul
  6. create a file “ /etc/consul.d/server/config.json ” and append the following data in it:

 

Note: You have to follow the above steps for rest of the servers excepts the client and adjust the IP addresses of servers accordingly in start_join.

Login to Client

  1. Repeat all the steps of Consul Installation.
  2. adduser consul
  3. mkdir -p /etc/consul.d/client
  4. mkdir /var/consul
  5. chown consul:consul /var/consul
  6. create a file “ /etc/consul.d/client/config.json ” and append the following data in it:

Now we are finished with the configuration part let’s get the cluster started:

Go to bootstrap server and execute the command below:

And for starting the servers just go to the servers and execute the command below:

Finally we are going to start the client:

If everything goes fine then you can see the list of members on each node by executing the command below:

On Client Node:

On Server Node:

On Bootstrap Server:

As you can see all the servers are connected to the bootstrap server. Now we can shutdown the bootstrapped Consul instance and then re-enter the cluster as a normal server.

To do this, hit Ctrl+C in the bootstrapped server’s terminal and start the consul service like you did with the rest of the servers. This will cause the previously-bootstrapped server to join the cluster with unelevated privileges, bringing the cluster into its final state.

Parts of logs:

On any of the node you can run this command to see the cluster members:

Parts of logs  and members list , when you connect bootstrap server as a normal server as you can see in the output of the consul members command below:

Logs:

Member list:

 

Network Tests

We use the ‘consul rtt’ command to test the network round-trip-time between the nodes of this cluster.

RTT between ip-10-0-20-100 & ip-10-0-20-101

Average is 0.7434 ms

RTT between ip-10-0-20-100 & ip-10-0-20-103

Average is 0.8756 ms

RTT between ip-10-0-20-100 & ip-10-0-20-104

Average is 0.8218 ms

As we can see, the RTT is mostly uniform since it’s a single subnet and the nodes can be configured as part of the same Consul datacenter. This is the simplest setup one can have but it also does not have much high availability as it cannot span multiple availability zones.

In the next post we will explore a cluster spread over multiple subnets in the same VPC.