Introduction

This post is part of a series on implementing a CI/CD pipeline. Please refer to this post for an overview and the table of content. In the first part of service discovery I discussed what service discovery is and how it can be implemented. In this post I want to show how we can configure a system to use consul for service discovery.

 Consul Master

Consul knows the concept of a master and an agent. The master node is where the data is stored. The agent works as a proxy on the node where it is installed and only forwards all calls to the master. In a production system everything has to be redundant. Thus we need to have at least 2 master nodes. Please note that if I am talking of a node this is either a physical server or a VM. Normally we would want 3 master nodes. Consul master is NOT recommended to run on Windows at this time, thus we will run all our consul masters on Linux. We will run consul inside a Docker container on Linux. Of course we need to run the 3 instances of consul master on 3 different nodes (that is not just 3 container instances on the same node), but that should be obvious. How can we do this?

Since we are running Consul in a Docker container we want to use Docker Swarm to create and manage a cluster of nodes. I am assuming you have installed Docker Toolbox on your system. If not, then download it from here.

Creating a cluster

Please refer to the Docker documentation here for the details about how to create a multi-host cluster. In this section I just want to give a brief summary of the steps needed and show the commands that I use to create such a cluster of nodes. First we need to have a key-value store that contains all the data about the cluster. We use Consul for this. Having VirtualBox installed on our system we can use Docker-Machine to create a VM for this key-value store. Open the Docker Quickstart console. We can use this command

This will create a new VM called my-keystore. You can run the Oracle VM VirtualBox GUI to see the new node as it is created and started. Once the VM is up and running we can download and run the consul Docker container with the following command

To double check whether or not the container is running as expected we can used the docker ps command.

We can then use the ID of the container to also look into the log of the container using

where {container-id} is the ID of the container. Tip: We only need to provide the first few characters of the ID for it to work. In our case it would be: docker logs 933c.

Now we can start to create VMs that represent the nodes of our cluster. We can use this command to generate the cluster master node which at the same time represents our swarm that we want to build

We want to add 2 more VMs to the cluster (you can add as many as you want…) using the following commands

OK, now we’re all set. Now we can list all nodes to confirm that they are up and running as expected.

We should see this:

We can use this command to set our environment to the Swarm master

Having done so we can use the docker info command to view the details of our cluster.

Running the consul master containers

Now let’s run an instance of Consul Master on each of the 3 nodes of our Swarm. Let’s start with the first node which happens to be the Swarm Master. First we need to find out what is the private IP-address of the node as well as the bride IP-address used for the swarm. To do this we can e.g. ssh into the node and run the command ifconfig.

we should see something in the line of

The bridge IP-address we can find under the docker0 section. The private IP-address is in the eth1 section. We can do that for the other nodes too. The bridge IP-address will be the same for all nodes but the private IP-address will change. In my case the other two nodes have 192.168.99.108 and 192.168.99.109.

Make sure you have set the environment to the master node and then run this command

This will run the first instance of Consul Master. We indicate that we want to create a cluster of 3 Consul Master nodes with the parameter -bootstrap-expect 3. Note how we use the internal IP-address 192.168.99.103 to publish ports 8xxx for tcp and udp and the bridge IP-address 172.17.0.1 to publish port 53 for udp. We are also using the internal IP-address to advertise.

Now switch the environment to the my-demo1 node and run

Note how we again use the private IP-address to publish ports 8xxx and the bridge IP-address for port 53. We also use the bridge IP-address to tell this second Consul node which Consul cluster/network to join.

Finally switch the environment to the my-demo2 node and run

Now if we look into the logs of the Consul Master on the node my-demo0 we should see something in the line of

#

Consul Agent

Now we want to run a Consul Agent directly on the Windows host. Since Docker does not yet run on Windows we have to install the binaries directly on Windows. Download and install the Consul Agent e.g. using Chocolatey as follows

Create a config file calle consul-Config.json in the folder c:\ProgramData\consul\config with the following content

Note that the IP-address corresponds to the publicly visible IP-address of my Windows host. Now start the Consul Agent as Windows service using this command

and then join the agent to the Consul network using this command

Now let’s have a look into the log files to see whether or not everything is OK. We can find the log for the Consul Agent on our Windows Host in the folder c:\ProgramData\consul\logs. Open the file consul-output.log. We should see something along the line of

Note how the agent has been added to the network of the 3 existing Consul Master nodes.

If we look at the tail of the log of one of the Consul Master nodes we should see something like

Which confirms that the agent (in my case DESKTOP-BFQ652M which happen to be the hostname of my Laptop) has successfully joined the network.

We can now use the UI to analyze what’s going on in our Consul network. Open the browser at http://localhost:8500/ui and you should see something like

Summary

I have shown in detail how to setup a Consul network with 3 master nodes running in Docker on Linux as well as a Consul Agent running on my Windows host. In the next post I will show how to register services with consul and how to monitor their health.