As part of a project we are doing, we decided to have a proper and low maintainance continuous deployment pipeline. We just wanted something that works.

The whole pipeline is depicted below:

CI Flow

GitHub Structure

We are using GitHub for our version control system. You can use any other like BitBucket or GitLab. We have one branch per environment:

Note: This structure is for demosntration purposes only. Branch based deployment is a bad practice for many reasons including increasing code divergence.

  • release for production release
  • test.1, test.2, … for our tests

We have a webhook to our CircleCI to trigger the build as soon as someone push code to the GitHub repository regardless of the branch. You may want to create a notification to your Slack channel as well but we found it very noisy.

CircleCI

We investigated few build systems. What we wanted was a system with good support for Containers as well as AWS integration. CircleCI seems like a very well-documented one with a proper pricing scheme.

I got ideas from lots of different sources few months back of writing this. I cannot find most of them at the moment, but I will reference those I can find.

This is what we want to happen:

  • For all branches, we need to build the image using the Dockerfile and then run the tests. At the end, we want to send the result of the build to our slack channel.
  • For specific branches, we need to push the image to AWS ECR, update ECS task definition, and finally update the service definition to use the latest task definition.
  1. Create a project in CircleCI, and select the GitHub repository you want to build.
  2. In the project setting > Permissions > AWS Permissions add your AWS key and secret. These are being used when we run commands using aws cli.
  3. Create a circle.yml file in the root of your project. CircleCI picks up this file and configures itself.
machine:
    node:
        version:
            0.10.33
    services:
        - docker

checkout:
    post:
        - chmod +x ./fix_dockerfile.sh && ./fix_dockerfile.sh
        - chmod +x ./deploy.prod.sh
        - chmod +x ./deploy.test.1.sh

dependencies:
    post:
        - sudo pip install --upgrade awscli
        - sudo apt-get update && sudo apt-get install jq
        - curl -L -o ~/bin/jq https://github.com/stedolan/jq/releases/download/jq-1.5rc1/jq-linux-x86_64-static && chmod +x ~/bin/jq
        - curl -L https://github.com/docker/compose/releases/download/1.5.0/docker-compose-`uname -s`-`uname -m` > ../bin/docker-compose && chmod +x ../bin/docker-compose
        - docker build --rm=false -t acmeinc/sample-api:$CIRCLE_SHA1 . | cat
test:
    override:
        - npm test

deployment:
    prod:
        branch: release
        commands:
            - ./deploy.prod.sh
    test:
        branch: test.1
        commands:
            - ./deploy.test.1.sh

Some notes about out circle.yml file:

  • Line 1..6: It tells that this app using node 0.10.33 and we need a docker container.
  • Lie 8..12: We are using different dockerfiles for different environments. After CircleCI loads the specific revision, it runs the fix_dockerfile.sh. It simply renames the specific file based on the branch name. i.e. ./Dockerfile.prod to ./Dockerfile. Then it gives deploy scripts execute premission. They will run at the end of the build process.
#!/usr/bin/env bash
set -e
set -u
set -o pipefail
fix_name() {
    DOCKERFILE_NAME="Dockerfile"
    if [[ $CIRCLE_BRANCH = 'release' ]]; then 
        DOCKERFILE_NAME="$DOCKERFILE_NAME.prod"; 
    elif [[ $CIRCLE_BRANCH = 'test.1' ]]; then
        DOCKERFILE_NAME="$DOCKERFILE_NAME.test.1"
    else
        DOCKERFILE_NAME="$DOCKERFILE_NAME.dev"
    fi;

    mv -f ./$DOCKERFILE_NAME ./Dockerfile
}

fix_name
  • Line 16..19: Installs awsclient and some other dependencies we need for deploy process.
  • Line 20: Builds the docker image with acmeinc/sample-api name and tags it with random hash of the current build.
  • Line 23: Runs tests.
  • Line 25..33: Based on the branch name runs the deployment script.

I changed this script a little bit.

#!/usr/bin/env bash

set -e

JQ="jq --raw-output --exit-status"

deploy_image() {
    # get the authorization code and login to aws ecr
    autorization_token=$(aws ecr get-authorization-token --registry-ids $account_id --output text --query authorizationData[].authorizationToken | base64 --decode | cut -d: -f2)
    docker login -u AWS -p $autorization_token -e none https://$account_id.dkr.ecr.us-east-1.amazonaws.com
    docker tag acmeinc/sample-api:$CIRCLE_SHA1 $account_id.dkr.ecr.us-east-1.amazonaws.com/sample-api:$CIRCLE_SHA1
    docker push $account_id.dkr.ecr.us-east-1.amazonaws.com/sample-api:$CIRCLE_SHA1
}

make_task_def() {
    task_template=$(cat ecs_taskdefinition.json)
    task_def=$(printf "$task_template" $CIRCLE_SHA1)
    echo "$task_def"
}

register_definition() {

    if revision=$(aws ecs register-task-definition --cli-input-json "$task_def" --family $family | $JQ '.taskDefinition.taskDefinitionArn'); then
        echo "Revision: $revision"
    else
        echo "Failed to register task definition"
        return 1
    fi

}

deploy_cluster() {
    
    
    make_task_def
    register_definition
    

    if [[ $(aws ecs update-service --cluster $cluster_name --service $service_name --task-definition $revision | \
                   $JQ '.service.taskDefinition') != $revision ]]; then
        echo "Error updating service."
        return 1
    fi

    for attempt in {1..30}; do
        if stale=$(aws ecs describe-services --cluster $cluster_name --services $service_name | \
                       $JQ ".services[0].deployments | .[] | select(.taskDefinition != \"$revision\") | .taskDefinition"); then
            echo "Waiting for stale deployments:"
            echo "$stale"
            sleep 5
        else
            echo "Deployed!"
            return 0
        fi
    done
    echo "Service update took too long."
    return 1

}

account_id=[aws_account_id]
family=acmeinc-api
service_name=acmeinc-api-srv

deploy_image
deploy_cluster

The above script basically:

  • Tags and pushes the container image to ECR
  • Updates the AWS ECS task definition to use the new image version that is $account_id.dkr.ecr.us-east-1.amazonaws.com/sample-api:$CIRCLE_SHA1
  • Registers the new task definition and gets the revision
  • Updates the service with the new task definition revision
  • Waits for the new service to be deployed to the cluster

Update: Sample task definition file:

{ 
     "containerDefinitions": [
        { 
            "name": "datacontainer",  
            "mountPoints": [
                {
                    "sourceVolume": "volume-0", 
                    "readOnly": false, 
                    "containerPath": "/app/acmeinc-api/log/"
                }
            ], 
            "image": "<aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/acmeinc-api-datavolume:latest", 
            "essential": false, 
            "portMappings": [], 
            "entryPoint": [], 
            "memory": 512, 
            "command": [], 
            "cpu": 1, 
            "volumesFrom": []
        }, 
        {
            "environment": [], 
            "name": "acmeinc-api", 
            "links": [], 
            "mountPoints": [], 
            "image": "<aws_account_id>.dkr.ecr.us-east-1.amazonaws.com/acmeinc-api:%s", 
            "essential": true, 
            "portMappings": [
                {
                    "protocol": "tcp", 
                    "containerPort": 8080, 
                    "hostPort": 8080
                }
            ], 
            "entryPoint": [], 
            "memory": 512, 
            "command": [], 
            "cpu": 1, 
            "volumesFrom": [
                {
                    "sourceContainer": "datacontainer"
                }
            ]
        }
    ],
    "volumes": [
        {
            "host": {
                "sourcePath": "/app/acmeinc-api/log/"
            }, 
            "name": "volume-0"
        }
    ]
}