Squadron Documentation¶

Multiple ways to get your code¶

With Squadron, you can download binaries, download tarballs of your code, grab your source with git. With apt, virtualenv, and npm support, you can get your code’s dependencies.

After grabbing your code, Squadron can run Built-in Tests and use Atomic Deployments to deploy your code.

Easy dependency management¶

Squadron supports apt, npm, and pip via virtualenv, and we’re always adding more. Adding a new package to install is super easy, and you can have different packages in different Environments.

Easy Templating¶

Squadron uses a simple templating library called Quik to write configuration templates. They look like this:

[log]
debuglog = @loglevel rotatinglog @logdir 5000 5
#if @output:
outputlog = DEBUG stderr
#end

This is used to write out your configuration files for your software. Values are set through defaults and service configurations and can be different for different Environments.

Atomic Deployments¶

Did you know that deploying your code takes time? And during that time your users might get inconsistent results? Why bother with that headache. Deploy your code atomically!

Squadron can easily deploy your code atomically via a symlink so that its either all the new version or doesn’t reflect any changes at all.

Environments¶

You want to have a development and production environments? Easy. You want to have six different QA environments? Done. You need two production environments for A/B testing? Trivial.

Environments are easy with Squadron.

Built-in Tests¶

Squadron has two types of built-in tests: automatic tests you don’t have to write, and an easy framework for testing your deployed code.

If you write a JSON file as part of a template, Squadron will check to make sure it’s a valid JSON file. In the future, XML files written out by templates will be similarly checked, and if they have a schema, that will be verified.

The second type of tests are the kind you write because you know your service. Do you write a PHP script to hit a URL? Or a simple bash script to check if your service is running? Whatever you want, we’ll run it.

Rollback¶

What if your deployment goes wrong?

With traditional configuration management software, it’s difficult or impossible to rollback to exactly how it was before the deployment. With Squadron, that’s built-in.

No programming¶

And, maybe best of all, there’s no programming involved. There’s no Domain Specific Language to learn. It’s all just config that is rigorously checked by Squadron when you run it.

No programming means less testing and little to no debugging so you can get what you want done faster and easier.

Templating¶

Squadron takes whatever files you have in root/ and deploys them to the correct directory (which in this):

$ cd services/website/0.0.1
$ tree -F root
root/
    ├── main~git
    └── robots.txt~tpl

So we’ve got two strange looking filenames. The tilde (~) means that Squadron will apply that handler to that file. The ‘~tpl’ extension is how you make files via a template.

Squadron uses the Quik templating library, so robots.txt~tpl will look something like this:

User-agent: *
#for @d in @disallow:
Disallow: @d
#end
Allow: /humans.txt

So the variable @disallow, which is an array, is looped over and so there are as many Disallow directives as elements in the array.

main~git looks like this:

{
    "url":"https://github.com/cxxr/example-squadron-repo.git",
    "refspec":"@release"
}

Squadron will clone this repo when it runs, checkout the refspec simple (which is a branch, a tag, or a hash) and place it in the ‘main’ directory. The variable '@release‘ will be replaced by whatever we set that to later

Configuration¶

How do all those values get set? They’re set in two ways.

The first is from the service configuration for each environment. Back in the top level of the Squadron directory, there’s a directory called config/. In it are your environments.

Environments are distinct places you can deploy your code to that don’t interact with each other. This allows you to have multiple testing environments that don’t affect your customers. Let’s make a development environment for our website:

$ cd -
$ ls
config/ services/ nodes/ libraries/
$ squadron init --env dev

Now there’s a file called config/dev/website.json, which is prepopulated with the latest version number. Let’s add the disallow config so the file looks like this:

{
    "base_dir": "/var/www",
    "config": {
        "disallow":["/secret/*","/admin/*"],
        "release":"master"
    },
    "version": "0.0.1"
}

The “version” field tells Squadron which service description version to use. Different environments can use different service description versions at the same time.

The “config” field is a JSON object that will be given to your service. These fields can be used in templates. If you have config that is often the same between environments, you can put it in another place.

The “base_dir” field tells Squadron where the root/ directory should be written to. Since we’re just deploying files to our web root, it’s /var/www.

The second way in which these values are set is via defaults.json. This file can be used to set default values in case none are set. Keys are the key in question, and the values are the values you would set in the config.

An equivalent defaults.json for our website would be:

{
    "disallow":["/secret/*","/admin/*"]
}

Schema¶

Squadron includes one very useful file with every service description called services/website/0.0.1/schema.json. This is a JSON schema describing the configuration that your service accepts. For our service it looks like this:

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "type" : "object",
    "properties" : {
        "disallow" : {
            "description" : "a list of disallow directives",
            "type" : "array",
            "items": {
                "type": "string"
            },
            "uniqueItems": true
        },
        "release" : {
            "description" : "what to checkout from the git repository",
            "type" : "string"
        }
    },
    "required": ["disallow", "release"]
}

This allows you to be sure that you passed in the correct types of input in your config files and in your defaults. If you don’t supply a JSON Schema, everything will still work, but it won’t be checked, either.

You can do some fairly advanced things with JSON Schema, such as regular expression matching. With this you could ensure that “release” met some tag pattern or similar.

New root¶

Let’s make our new root directory:

$ cd services/website/1.0.0/root
$ mkdir -p var/www/
$ mv main~git var/www/
$ mv robots.txt~tpl var/www/

You can see how this reflects our new base_dir of root. It would also be nice if we released our web root atomically so that if anyone happens to load it while we’re copying over, they don’t get half new and half old assets. Fortunately, this is really easy with Squadron:

$ cat > config.sq
var/www/ atomic:true user:nobody group:nobody

The config.sq file in the root directory of a service is special. It’s not copied to your base_dir, but instead configures some metadata, such as setting the user, group, or mode for a file or directory. For more information, see the Config.sq section of the User Guide.

What we’ve done here is to tell Squadron to do an atomic deploy of var/www/, which means it will use a symbolic link from /var/www/ to Squadron’s deployment directory.

Apache module¶

We also need to make sure that PHP is installed:

$ cd ..
$ cat > state.json
[
    {
        "name":"apt",
        "parameters":["apache2", "libapache2-mod-php5"]
    }
]

Now we need to run a2enmod when this is installed. We actually need to set up two files for this: actions.json and react.json.

The file actions.json describes the possible actions that can take place. These are commands that are run. Sometimes restarting the service, sometimes starting it. Ours will look like this:

{
    "run a2enmod php": {
        "commands": ["a2enmod php5", "/etc/init.d/apache2 restart"],
    },
    "start" : {
        "commands" : ["/etc/init.d/apache2 start"]
    },
    "reload" : {
        "commands" : ["/etc/init.d/apache2 reload"],
        "not_after" : ["start", "restart"]
    },
    "restart" : {
        "commands" : ["/etc/init.d/apache2 restart"],
        "not_after" : ["start"]
    }
}

So we have four actions. Three are easy enough to understand: they control the running of the service. Starting apache, reloading it, and restarting it. The not_after property means that if there are several actions to run for a deployment, that these should not be run after successful invocations of those. This will be more clear after understanding react.json.

The file react.json describes how to react to various events. It gives criteria for the events and then which actions to execute. Ours looks like this:

[
    {
        "execute": ["run a2enmod php"],
        "when" : {
            "not_exist": "/etc/apache2/mods-enabled/php5"
        }
    },
    {
        "execute": ["start"],
        "when" : {
            "command": "/etc/init.d/apache2 status",
            "exitcode_not": 0
        }
    },
    {
        "execute" : ["reload"],
        "when" : {
            "files" : ["*.conf", "*/conf.d/*"]
        }
    }
]

For a complete description of actions and reactions, see Action and reaction in the User Guide.

Let’s do it:

$ sudo squadron apply -n dev
Staging directory: /var/squadron/tmp/sq-8
Processing apache2, libapache2-mod-php5 through apt
Applying changes
Running action website.run a2enmod php in reaction {u'execute': [u'website.run a2enmod php'], u'when': {u'not_exist': [u'/etc/apache2/mods-enabled/php5']}}
Module php5 already enabled
* Restarting web server apache2
    apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1 for ServerName
... waiting apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1 for ServerName   [ OK ]
Apache2 is running (pid 2332).
Successfully deployed to /var/squadron/tmp/sq-8
===============
Paths changed:

New paths:
    website/var/www/main/LICENSE
    website/var/www/main/index.html
    website/var/www/main/README.md
    website/var/www/robots.txt
$ ls -l /var/www
lrwxrwxrwx 1 root root 39 Jan 01 00:00 /var/www -> /var/squadron/tmp/sq-8/website/var/www/

And navigating to http://localhost works!

Testing¶

An important part of deploying software is making sure it’s correct. For our purposes, we want to check that PHP is working and that Apache was set up correctly.

In Squadron, Tests are located in the service’s tests directory. Let’s make one now:

$ mkdir -p services/website/1.0.0/tests
$ cat > services/website/1.0.0/tests/check_php.sh
#!/bin/bash

while read line; do
    true
done

OUTPUT=`curl http://localhost/main/test.php 2>/dev/null`

if [ "$?" -eq "0" ]; then
    if [[ $OUTPUT == *php* ]]; then
        echo "PHP not enabled"
        exit 1
    fi
else
    echo "Couldn't connect"
    exit 1
fi

Tests must read in the JSON object passed via standard in. For our test, we don’t care about the configuration, so we just throw it away.

We then test that the connection worked via the exit code flag $?. If curl was successful, we check to make sure the output didn’t have the string “php” in it, which would indicate that PHP wasn’t configured properly.

Almost done. We just need to make sure this test is executable and that curl is installed:

$ chmod +x services/website/1.0.0/tests/check_php.sh
$ cat > services/website/1.0.0/state.json
[
    {
        "name":"apt",
        "parameters":["apache2", "libapache2-mod-php5", "curl"]
    }
]

And now we’re done. Let’s run it:

$ sudo squadron apply -n dev
Staging directory: /var/squadron/tmp/sq-11
Processing apache2, libapache2-mod-php5, curl through apt
Running 1 tests for website v1.0.0
Nothing changed.

dir¶

The ‘dir’ extension creates an empty directory of that name.

Contents

None

download¶

The ‘download’ extension downloads a file over HTTP.

Contents

A single line of the HTTP endpoint with an optional SHA256 prefix hash of the file. Will have a template applied to it, so variable substitution and logic is possible.

Examples:

http://www.example.com/filename.ext a7898bc

or:

http://no.sha256.here.com/file.to.download.txt

extract¶

The ‘extract’ extension downloads and extracts tarballs and zip files.

Contents

The extract extension handler takes a JSON object like this:

{
    "url": "http://www.example.com/dir/file.tar.bz2"
}

or with manually specifying the type:

{
    "url": "https://www.example.com/dir/filename_without_ext",
    "type": "tar.gz"
}

The extract handler can also copy files out of the tarball:

{
    "url": "https://www.example.com/dir/file.zip",
    "persist": false,
    "copy": [
        {
            "from": "test*",
            "to": "/etc/test/"
        },
        {
            "from": "dir/file.txt",
            "to": "../file.txt"
        }
    ]
}

The full list of supported fields is described in the following table.

The copy objects are described in the following table.

git¶

The ‘git’ extension clones git repositories.

Contents

A JSON object with properties such as “url”. Will have a template applied to it, so variable substitution and logic is possible.

Examples:

{
    "url":"https://github.com/gosquadron/squadron.git"
}

or:

{
    "url":"git@github.com:gosquadron/example-squadron-repo.git",
    "refspec":"experimental"
}

or even:

{
    "url":"git@github.com:gosquadron/example-squadron-repo.git",
    "refspec":"@release",
    "sshkey":"ssh_keys/deploy1",
    "args":"--depth=2"
}

The last example requires that the ssh_keys/deploy1 resource exists and is a private ssh key. See the Resources section for more information. It also does a shallow clone of the git repository via the –depth argument.

The properties allowed in the object are described in the following table:

tpl¶

The template extension simply applies a template to the given file.

Contents

The template is the content.

Example:

<html>
    <body>
        <h1>Hello, @user!</h1>
#for @p in @paragraphs:
        <p>@p</p>
#end
    </body>
</html>

virtualenv¶

Creates a Python virtualenv. The virtualenv and pip commands must be available and in the current user’s PATH. Run through a template so variable substitution is possible.

Contents

The contents of this file are passed to pip as if they were a requirements.txt file.

Example:

Flask==@versions.flask
Jinja2==2.6
Werkzeug==0.8.3
certifi==0.0.8
chardet==1.0.1
distribute==0.6.24
gunicorn==0.14.2
requests==0.11.1

How to write a library¶

In the libraries directory of your Squadron repository, you can place a Python module.

The Python module should expose three functions:

def schema():
    return {}

def verify(inputhashes):
    return []

def apply(inputhashes, dry_run=True):
    return []

The schema function should return the Python representation of a JSON schema. It describes one object passed into the verify function.

The verify function takes a list of objects (of the type described in the schema). It then returns a list of objects that are not already in the state specified.

The apply function takes the list of objects that failed verification (weren’t yet in the state they were supposed to be in) and a boolean dry_run. It returns a list of objects that couldn’t be applied.

Included libraries¶

Some libraries are included with Squadron so you don’t have to write them yourself.

The state.json file is passed through the template engine before being executed, so you can embed logic and variables within it like you would normally.

[
    {
        "name":"apt",
        "parameters":["screen","tmux"]
    }
]

[
    {
        "name":"group",
        "parameters":[
            {
                "name": "newgroup"
            },
            {
                "name": "specificgroup",
                "gid": 555,
                "system": true
            }
        ]
    }
]

[
    {
        "name":"user":
        "parameters":[
            {
                "username": "newuser"
            },
            {
                "username": "specificuser",
                "shell":"/bin/bash",
                "homedir":"/users/specificuser"
                "realname":"Specific User"
            },
            {
                "username":"windows",
                "uid":666,
                "system":true
            }
        ]
    }
]

Actions¶

Actions are described in actions.json in each service. An action has a name, a list of commands to run, and a list of actions to not run this one after.

Here’s what one might look like:

{
    "start" : {
        "commands" : ["/etc/init.d/service start"]
    },
    "reload" : {
        "commands" : ["killall -HUP service"],
        "not_after" : ["start", "restart"]
    },
    "restart" : {
        "commands" : ["/etc/init.d/service restart"],
        "not_after" : ["start"]
    }
}

So this service has three actions. The start command starts up the service. The restart command restarts it, but only if the start command didn’t just succeed. This way you can avoid restarting a service immediately after starting it.

Here are the possible fields to put in an action:

Reactions¶

Reaction trigger actions in this service or other services based on files being created or modified. The reactions are described in react.json in each service.

One might look like this:

[
    {
        "execute": ["start", "apache2.restart"],
        "when" : {
            "command": "pidof service",
            "exitcode_not": 0
        }
    },
    {
        "execute" : ["restart"],
        "when" : {
            "files" : ["mods-enabled/*"]
        }
    },
    {
        "execute" : ["reload"],
        "when" : {
            "files" : ["*.conf", "conf.d/*"]
        }
    }
]

The first reaction starts this service and restarts another service called apache2 when it’s not running.

The second reaction restarts this service if there are any modules created or modified. You can use ‘files-created’ or ‘files-modified’ to narrow this scope.

The third reaction reloads this service when any of the config files change.

The executing actions must be defined in actions.json or an error will be raised.

Here is a list of fields the top level reaction object can contain:

Here is a list of fields that a when object can contain:

Daemon¶

This section configures the daemon.

Squadron¶

This is the main configuration section, which configures most of Squadron’s settings. It’s used by the daemon and by running Squadron in standalone mode (via the check, apply, or init commands).

Status¶

This section controls the sending of status updates to a Squadron state server.

Log¶

This section is a bit different, as you can enter as many lines as you want here so long as they follow the following format defined in the example:

debugonly = DEBUG file /tmp/log

Which means:

• debugonly - Just an identifier. This log name must be unique.
• DEBUG - Level to log must match one of Python’s log levels.
• file - Type of log, in this case this is a simple file log.
• /tmp/log - Parameter(s) for the type of log, which is, in this case, the file to log to.

Squadron supports three types of logs:

file:

• expects file to log to as parameter

stream:

• expects stdout or stderr as the parameter

rotatingfile:

• file to log to
• max file size in bytes
• max number of files to backup

Example of rotating file located at /var/log/squadron/log, which rotates every 5000 bytes and keeps two files in backup:

rotate = DEBUG rotatingfile /var/log/squadron/log 5000 2