Monday, July 6, 2015

Setting Up a CoreOS Cluster with Fleet and Vagrant

Prerequisites: You'll want to be familiar with setting up VirtualBox with Vagrant and have Homebrew installed. This guide was written for Mac OS X users.

Installing Dependencies

  • VirtualBox
  • fleetctl
  • Vagrant


On our Mac OS X host machine, we need to make sure we have the above dependencies installed. Since we're already familiar with VirtualBox after going through the prerequisites, we install fleetctl like so:

# install fleetctl using brew
brew update
brew install fleetctl

# or from source
wget && unzip
sudo cp fleet-v0.10.2-darwin-amd64/fleetctl /usr/local/bin/

Note: Make sure you update the path to reflect the current version of fleet against their releases page.



Now, we're going to clone the official CoreOS Vagrant setup and generate a unique discovery token to share across all of your instances. We'll need a new token every time we create a cluster or your instances won't be able to register.

# clone coreos' vagrantfile repo
git clone
cd coreos-vagrant

# allow fleetctl to speak to the Vagrant vm...
# ...over SSH
ssh-add ~/.vagrant.d/insecure_private_key

# create a user-data file with a new token
DISCOVERY_TOKEN=`curl -s` && perl -p -e "s@#discovery: $DISCOVERY_TOKEN@g" user-data.sample > user-data

Note: In case you wanted to generate the token, manually, you can just as easily run curl -s


The user-data file we generated off of user-data.sample is your cloud-config, so get familiar with it. When you open up the user-data file, you should see that your token was inserted, ready to run etcd.


# copy and rename the sample file
cp config.rb.sample config.rb

# edit the file
vi config.rb

Now, you'll want to uncomment and update the following lines:

# Size of the CoreOS cluster created by Vagrant

# Official CoreOS channel from which updates should be downloaded

Now, we can spin up our cluster.

# spin up the instance
vagrant up

# list the status of the running machines
vagrant status

Current machine states:

core-01                   running (virtualbox)
core-02                   running (virtualbox)
core-03                   running (virtualbox)

Running a Docker Container on CoreOS

Creating a SystemD Unit File

Now, were going to create a systemd unit file that will tell systemd how to start and stop different containers on these three instances. If you need a refresher on Docker commands, check out this article. Create the following file on your host machine, and naming it hello-01.service. Then, duplicate this files twice, with the hello-02.service and hello-03.service filenames. Make sure to increment the values in the file, as well. I've highlighted the lines, to make it easier to identify.

[X-Fleet] is a CoreOS-specific extension to the systemd unit file syntax. Here, we're instructing fleet not to run any of these service on the same instance. This is handy for when you was to ensure redundancy across nodes. We'll get into more specific configurations in a future post.

Now, let's also spin up an NGiNX container, for good measure. This time, we're going to instruct Fleetctl to deploy this container on core-01. Create the following file and name it nginx.service.

Note: The CoreOS documentation says to "...not run docker containers with -d as this will prevent the container from starting as a child of this pid. systemd will think the process has exited and the unit will be stopped"[1].

Listing Our Machines

fleetctl list-machines

8354c7d4...  -
90fd1e17...  -
d6754bc5...  -

Submitting the Hello World Services

# submit the service
fleetctl submit hello-01

# verify it was submitted
fleetctl list-unit-files
UNIT                    HASH    DSTATE          STATE     TARGET
hello-01.service ac0cb1e inactive inactive -

# start the service
fleetctl start hello-01
Unit hello-01.service launched on 48bd0f4a.../

# list running units
fleetctl list-units
UNIT                    MACHINE                         ACTIVE  SUB
hello-01.service 48bd0f4a.../ active running

# view logs
fleetctl journal hello-01
-- Logs begin at Tue 2015-07-07 01:16:53 UTC, end at Tue 2015-07-07 02:17:22 UTC. --
Jul 07 02:17:13 core-02 docker[1728]: Hello World 1
Jul 07 02:17:14 core-02 docker[1728]: Hello World 1

Now, repeat the same steps for hello-02.service and hello-03.service.

Submitting the Nginx Service

Because we made the MachineOf=hello-01.service in our nginx.service file, the NGiNX container will run on the same machine as hello-01.service. For this session, that ended up being core-02, but it may be one of the other machines when you run these commands.

# submit the service
fleetctl submit nginx

# verify it was submitted
fleetctl list-unit-files
UNIT            HASH    DSTATE    STATE     TARGET
nginx.service   0574bed inactive  inactive  -

# start the service
fleetctl start nginx.service
Unit nginx.service launched on 8354c7d4.../

# list running units
fleetctl list-units
UNIT          MACHINE                   ACTIVE  SUB
nginx.service 8354c7d4.../  active  running

# view logs
fleetctl journal nginx.service

The NGiNX container is now running on core-02. You can verify this by navigating to in your browser. You're also able to ssh into the machine and run the basic, diagnostic Docker commands.

# ssh in
vagrant ssh core-02

# view running containers
docker ps
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS                           NAMES
6d0a3354a282        nginx:latest        "nginx -g 'daemon of   25 minutes ago      Up 25 minutes       443/tcp,>80/tcp   mynginx

Updating a Service

Run these commands if you make any changes to your *.service files.

fleetctl destroy [NAME]
fleetctl start [NAME]

In case you want to destroy the cluster, just run vagrant destroy -f. But remember, you need to generate a new token everytime you spin up a new cluster, so return to the Vagrant seciont above, and set a new DISCOVERY_TOKEN.

That's all we'll cover during this introductory guide. Check back, soon, for a follow up.


1. ^ CoreOS (Last Accessed: 06 July 2015). "Getting Started with systemd"

No comments:

Post a Comment