#travisci

[iplanz]

3 down 2 to go. On to ggplot #Rprogramming #buildingRpackages #TravisCI https://www.instagram.com/p/Bz28_lRHyuR/?igshid=1hadujqf1hore

[luca-tumedei]

Installing and setting up WordPress and Nginx for Travis CI tests.

Installing and setting up WordPress and Nginx for Travis CI tests.

You should probably set up Continuous Integration system I could go at length about why setting up a “Continuous Something” flow is something you should do, but I’d rather defer the theory and the introduction to this article on Nutcache blog as it provides a clear high-level view of the matter while also providing good starting points for each touched subject. In a way not dissimilar to what…

View On WordPress

[mlbors]

Setting up a workflow with Docker, GitHub, Travis and AWS

In this post, we are going to set up a workflow to test and deploy automatically a PHP app on Amazon Web Services Elastic Beanstalk.

The aim of this article is to set up a basic workflow using a few tools to automate our deployment process. First of all, we need to have an account on each of these platforms: GitHub, Docker Hub, Amazon Web Services (AWS) and Travis CI. We also need to have Docker and Git installed on our machine.

Basically, we are going to do the following things: use Docker to create our environment, develop our app, create a repository, commit and push everything to GitHub, let Travis CI do the tests, build a Docker image with our code and push everything to Elastic Beanstalk (EB).

Setting up the environment

To create our environment, we are going to use the PHP Docker Boilerplate from WebDevOps. We can clone the repository by simply use:

git clone https://github.com/webdevops/php-docker-boilerplate.git our-appt

Cloning webdevops/php-docker-boilerplate

We will just adjust a very few things. First, in the Dockerfile.production file, at the end of it, we are going to add this simple line:

COPY ./app/. /app/

Here, we are copying our app folder instead of sharing it with the host.

If we use another image, we may also need to expose ports and run a command when launching our container like so:

EXPOSE 80 EXPOSE 443 CMD ["supervisord"]

Initializing our project

We can now go to GitHub and create a new repository. After that, we can make our first commit:

git init git add . git commit –m "First commit" git remote add origin https://github.com/user/repository.git git push –u origin master

Initializing our repository

For our example, we create a really simple app using Composer. We must be sure to have the appropriate .gitignore file. A .dockerignore file could also be a good idea:

*.md .git* backup/* bin/* documentation/*

.dockerignore file example

We can now link our GitHub repository to Travis CI.

AWS / EB

We are now going to set a few things on AWS. First, we need to create a new application on EB. We can name it however we want. Then, we have to create new environment. Here, we need to select "Docker Multicontainers" for the platform option. We also need to enable VPC. We can then proceed.

When it is done, we need to assure that our instance profile can communicate with Amazon ECS. To be sure of this, we can attach the AWSElasticBeanstalkMulticontainerDocker policy to it. To set the instance profile, in the EB dashboard, we can go to Configuration > Instances settings. To attach a policy to a role, in the IAM dashboard, we can go to Roles, choose the concerned role and then attach the desired policy.

We also need to create a new user. If there is no available group, we have to create on first. When our user is created, we can request credentials. We need to be sure to keep the secret key somewhere because we won't be able to access it later.

Information we need

Before going any further, we need to be sure to have the following information:

Our app name (APP_NAME)

Our Docker username (DOCKER_USERNAME)

Our Docker repository name (DOCKER_REPOSITORY)

Our Docker password (DOCKER_PASSWORD - secure)

Our email adress used with Docker (DOCKER_EMAIL)

An image name (IMAGE_NAME)

The AWS bucket name (BUCKET_NAME)

Our deployment region (DEPLOYMENT_REGION)

The deployment bucket (DEPLOYMENT_BUCKET)

The deployment environment (DEPLOYMENT_ENV_NAME)

The deployment's id (DEPLOYMENT_ENV - secure)

Our AWS access key (AWS_ACCESS_KEY - secure)

Our AWS secret key (AWS_SECRET_KEY - secure)

Because we are using Travis CI, we can set these values in our Travis CI account in the settings sections of our project or put them in the .travis.yml file and encrypt the private ones with the following command (Travis CI gem need to be installed):

travis encrypt SOMEVAR=secretvalue --add

Docker Configuration

In case we would like to use a private repository on Docker Hub, we need to create a file named .dockercfg like so:

{ "https://index.docker.io/v1/": { "auth": "", "email": "" } }

.dockercfg file

The Amazon ECS container agent will use this file for authentication. Because Docker create a file with another format, we are going to feed this file later with the values that in the files generated by Docker.

Now we can create another file named Dockerrun.aws.json. This file will be used by EB to deploy our Docker containers. Here again, we are going to replace values in this file later.

{ "AWSEBDockerrunVersion": 2, "authentication": { "bucket": "", "key": ".dockercfg" }, "volumes": [ { "name": "storage", "host": { "sourcePath": "/var/data" } }, { "name": "app", "host": { "sourcePath": "/var/app/current/app" } } ], "containerDefinitions": [ { "name": "db", "image": "mysql:5.6", "essential": true, "memory": 512, "portMappings": [ { "hostPort": 3306, "containerPort": 3306 } ] "mountPoints": [ { "sourceVolume": "storage", "containerPath": "/var/lib/mysql" } ], "environment": [ { "name": "MYSQL_ROOT_PASSWORD", "value": "password" }, { "name": "MYSQL_DATABASE", "value": "my_db" } ] }, { "name": "app", "image": ":", "essential": true, "memory": 256, "portMappings": [ { "hostPort": 80, "containerPort": 80 } ], "links": [ "db" ], "mountPoints": [ { "sourceVolume": "app", "containerPath": "/var/www/html", "readOnly": true } ] } ] }

Dockerrun.aws.json file

We tell Travis CI what to do

Now, with the .travis.yml file, we can tell Travis CI what to do.

sudo: required language: php python: - "3.4" - "pypy-5.3.1" services: - docker before_install: # Install dependencies - gem update --system - sudo apt-get install -y python3.4 - sudo apt-get install --upgrade -y python-pip - sudo apt-get install jq - sudo pip install --user virtualenv # Create a virtual environment for AWS CLI - virtualenv my_py3 --python=/usr/bin/python3.4 - source my_py3/bin/activate - pip install --upgrade awscli - pip install --upgrade awsebcli # Set AWS information - aws configure set aws_access_key_id $AWS_ACCESS_KEY - aws configure set aws_secret_access_key $AWS_SECRET_KEY - aws configure set default.region $DEPLOYMENT_REGION - aws configure set metadata_service_timeout 1200 - aws configure set metadata_service_num_attempts 3 - aws configure list # Copy the docker-compose.production.yml file to docker-compose.yml file - cp docker-compose.production.yml docker-compose.yml # Build and create our containers - docker-compose up -d - docker ps -a before_script: # Install dependencies in the app container - docker-compose exec -T app composer self-update - docker-compose exec -T app composer install --no-interaction - docker-compose exec -T app composer dump-autoload -o script: # Run unit tests in the app container - docker-compose exec -T app vendor/bin/php-cs-fixer fix app --verbose - docker-compose exec -T app vendor/bin/phpunit --coverage-clover=coverage.xml after_success: # Stop containers and build our image - docker-compose stop - docker-compose build --no-cache # Send coverage information to Codecov (if needed) - bash <(curl -s https://codecov.io/bash) # Push image to Docker Hub and update EB environment - if [ "$TRAVIS_BRANCH" == "master" ]; then docker login -u="$DOCKER_USERNAME" -p="$DOCKER_PASSWORD"; docker tag $IMAGE_NAME $DOCKER_USERNAME/$DOCKER_REPOSITORY:$TRAVIS_BUILD_ID; docker push $DOCKER_USERNAME/$DOCKER_REPOSITORY:$TRAVIS_BUILD_ID; ./scripts/upload_image_to_elastcbeanstalk.sh $TRAVIS_BUILD_ID $DEPLOYMENT_BUCKET $DEPLOYMENT_ENV $APP_NAME $DEPLOYMENT_REGION $IMAGE_NAME $DEPLOYMENT_ENV_NAME $DOCKER_USERNAME $DOCKER_REPOSITORY $DOCKER_PASSWORD $DOCKER_EMAIL; fi notifications: email: false env: global: - APP_NAME=app_name - DOCKER_USERNAME=docker_username - DOCKER_REPOSITORY=docker_repository (here same as app_name) - IMAGE_NAME=image_name - BUCKET_NAME=bucket_name (here same as app_name) - DEPLOYMENT_REGION=us-east-2 (for example) - DEPLOYMENT_BUCKET=elasticbeanstalk-us-east-2-xxxxxxxxxxxx - DEPLOYMENT_ENV_NAME=env_name - DOCKER_EMAIL=docker_email - secure: some_secure_value - secure: some_secure_value - secure: some_secure_value - secure: some_secure_value

.travis.yml file

Let's recap what we are doing here:

We install the dependencies we need

We install AWS CLI in a virtual environment to avoid conflicts and set information

Create a docker-compose.yml file from the docker-compose.production.yml file

Start our containers

Install dependencies in the app container

Run tests in the app container

Build our image

If everything is alright and if we are on master branch, we push the image to Docker Hub and call our script to update the EB environment

EB environment

In our .travis.yml file, we call a script named upload_image_to_elastcbeanstalk.sh. Let's see what it is:

#! /bin/bash # Variables DOCKER_TAG=$1 DOCKERRUN_FILE="Dockerrun.aws.json" DOCKERCFG=".dockercfg" DOCKER_CONFIG="/home/travis/.docker/config.json" EB_BUCKET=$2 EB_ENV=$3 PREFIX="deploy/$DOCKER_TAG" APP_NAME=$4 DEPLOYMENT_REGION=$5 IMAGE_NAME=$6 DEPLOYMENT_ENV_NAME=$7 DOCKER_USERNAME=$8 DOCKER_REPOSITORY=$9 DOCKER_PASSWORD=${10} DOCKER_EMAIL=${11} DOCKER_IMAGE="$DOCKER_USERNAME/$DOCKER_REPOSITORY" # Generate dockercfg echo "::::: Creating .dockercfg file :::::" DOCKER_AUTH=($(sudo jq -r '.auths["https://index.docker.io/v1/"].auth' $DOCKER_CONFIG)) cat "$DOCKERCFG" \ | sed 's||'$DOCKER_AUTH'|g' \ | sed 's||'$DOCKER_EMAIL'|g' \ > $DOCKERCFG sleep 30 aws s3 cp $DOCKERCFG s3://$EB_BUCKET/.dockercfg sleep 30 echo "::::: Creating Dockerrun.aws.json file :::::" # Replace vars in the DOCKERRUN_FILE cat "$DOCKERRUN_FILE" \ | sed 's||'$EB_BUCKET'|g' \ | sed 's||'$DOCKER_IMAGE'|g' \ | sed 's||'$DOCKER_TAG'|g' \ > $DOCKERRUN_FILE sleep 30 aws s3 cp $DOCKERRUN_FILE s3://$EB_BUCKET/$PREFIX/$DOCKERRUN_FILE sleep 30 echo "::::: Creating new Elastic Beanstalk version :::::" # Run aws command to create a new EB application with label aws elasticbeanstalk create-application-version \ --region=$DEPLOYMENT_REGION \ --application-name $APP_NAME \ --version-label $DOCKER_TAG \ --source-bundle S3Bucket=$EB_BUCKET,S3Key=$PREFIX/$DOCKERRUN_FILE sleep 30 echo "::::: Updating Elastic Beanstalk environment :::::" aws elasticbeanstalk update-environment \ --environment-id $EB_ENV \ --environment-name $DEPLOYMENT_ENV_NAME \ --application-name $APP_NAME \ --version-label $DOCKER_TAG echo "::::: Removing file :::::" sleep 30 rm $DOCKERCFG rm $DOCKERRUN_FILE

upload_image_to_elastcbeanstalk.sh file

Let's recap what we are doing here:

We parse the file /home/travis/.docker/config.json and extract the "auth" value from it

We replace values in the .dockercfg and upload it on AWS S3

We replace values in Dockerrun.aws.json file and upload it on AWS S3

We create a new version for the application

We update the EB environment

We remove the configuration files

The end!

If everything is alright, our app will be deployed on AWS EB by the next push.

Test locally and fix problems

If we need to run our image locally, we can do the following thing:

docker pull username/repository:tag docker run --expose 80 -p 80:80 -it username/repository:tag

Or if we need to use sh:

docker run -ti username/repository:tag sh

To get console output from the instance, we can use the following command (awscli has to be installed; we may need to run aws configure first):

aws ec2 get-console-output --instance-id instance_id

Getting console output

We may also want to connect to our instance using SSH. To do so, we just have to use the following command (awsebcli has to be installed; we may need to run eb init first):

eb ssh --setup // If needed eb ssh environment-name

Using SSH

We can retrieve and display logs like so:

eb logs

Displaying logs

Further configuration and optimization

We may want to extend our configuration. For that, we can place some .config files in the .ebextentions folder. For example, we may need a script to clean unused images:

option_settings: - namespace: aws:elasticbeanstalk:command option_name: Timeout value: 1200 commands: docker_clean_containers: command: docker rm -v $(docker ps -a -q) ignoreErrors: true docker_clean_images: command: docker rmi $(docker images -q) ignoreErrors: true

Cleaning unused images - .ebextentions/docker.config

In our example, we also use a MYSQL container for our database. Using AWS RDS instead can be a reliable choice for production.

Conclusion

Setting up this whole process may seem to be quite a headache that involves many different tools. On the other hand, we have the opportunity to automate our deployment process while preserving control and security.

[mlbors]

Angular 2, Travis CI, Coveralls and Open Sauce

In this post, we are going to see how we can make an Angular 2 app work with Travis CI, Coveralls and Open Sauce.

In a previous post, we saw how to set up a PHP project using Travis CI, StyleCI and Codecov. We are now going to do quite the same, but with a small Angular 2 app. Our purpose here is to test our app under multiple environments on each commit.

So the first thing we need to do is to sign up for a GitHub account, then we will have access to Coveralls with that same account. The next thing is to open another account on Open Sauce. We are now ready to begin! Let's create our repository, commit and push!

To achieve our goal, it is important that our project runs with angular-cli. We are going to assume that is the case and that Git is also installed.

Dependencies

For a start, we need to install several dependencies with NPM. Let's do it like so in the terminal:

npm install karma-coverage karma-coveralls karma-firefox-launcher angular-cli-ghpages --save-dev

Install dependencies

With that last command, we installed karma-coverage that is responsible for code instrumentation and coverage reporting. karma-coveralls will help us to transmit the report to Coveralls. karma-firefox-launcher is a Karma plugin that will help us with our tests. Finally, angular-cli-ghpages will help us to deploy our app on GitHub Pages.

Karma

Now we need to set a few things in the karma.conf.js file that is included in the root of our folder. The file will look like so:

module.exports = function (config) { config.set({ basePath: '', frameworks: ['jasmine', '@angular/cli'], plugins: [ require('karma-jasmine'), require('karma-chrome-launcher'), require('karma-firefox-launcher'), require('@angular/cli/plugins/karma'), require('karma-coverage') ], client:{ clearContext: false // leave Jasmine Spec Runner output visible in browser }, files: [ { pattern: './src/test.ts', watched: false } ], preprocessors: { 'dist/app/**/!(*spec).js': ['coverage'], './src/test.ts': ['@angular/cli'] }, mime: { 'text/x-typescript': ['ts','tsx'] }, coverageReporter: { dir : 'coverage/', reporters: [ { type: 'html' }, { type: 'lcov' } ] }, angularCli: { config: './angular-cli.json', codeCoverage: 'coverage', environment: 'dev' }, reporters: config.angularCli && config.angularCli.codeCoverage ? ['progress', 'coverage'] : ['progress'], port: 9876, colors: true, logLevel: config.LOG_INFO, autoWatch: true, browsers: ['Chrome', 'Firefox'], singleRun: false }); };

karma.conf.js file

Karma is a test runner that is ideal for writing and running unit tests while developing the application.

Protractor

We are now going to configure Protractor for e2e testing. There is already a file for that in the root of our folder, but is only suitable for local tests with a single browser. Let's create a new one a folder called config. We will name that file protractor.sauce.conf.js and it will be like the following:

var SpecReporter = require('jasmine-spec-reporter').SpecReporter; var buildNumber = 'travis-build#'+process.env.TRAVIS_BUILD_NUMBER; exports.config = { sauceUser: process.env.SAUCE_USERNAME, sauceKey: process.env.SAUCE_ACCESS_KEY, allScriptsTimeout: 72000, getPageTimeout: 72000, specs: [ '../dist/out-tsc-e2e/**/*.e2e-spec.js', '../dist/out-tsc-e2e/**/*.po.js' ], multiCapabilities: [ { browserName: 'safari', platform: 'macOS 10.12', name: "safari-osx-tests", shardTestFiles: true, maxInstances: 5 }, { browserName: 'chrome', platform: 'Linux', name: "chrome-linux-tests", shardTestFiles: true, maxInstances: 5 }, { browserName: 'chrome', platform: 'macOS 10.12', name: "chrome-macos-tests", shardTestFiles: true, maxInstances: 5 }, { browserName: 'chrome', platform: 'Windows 10', name: "chrome-latest-windows-tests", shardTestFiles: true, maxInstances: 5 }, { browserName: 'firefox', platform: 'Linux', name: "firefox-linux-tests", shardTestFiles: true, maxInstances: 5 }, { browserName: 'firefox', platform: 'macOS 10.12', name: "firefox-macos-tests", shardTestFiles: true, maxInstances: 5 }, { browserName: 'firefox', platform: 'Windows 10', name: "firefox-latest-windows-tests", shardTestFiles: true, maxInstances: 5 }, { browserName: 'internet explorer', platform: 'Windows 10', name: "ie-latest-windows-tests", shardTestFiles: true, maxInstances: 5 }, { browserName: 'MicrosoftEdge', platform: 'Windows 10', name: "edge-latest-windows-tests", shardTestFiles: true, maxInstances: 5 } ], sauceBuild: buildNumber, directConnect: false, baseUrl: 'YOUR_GITHUB_PAGE', framework: 'jasmine', jasmineNodeOpts: { showColors: true, defaultTimeoutInterval: 360000, print: function() {} }, useAllAngular2AppRoots: true, beforeLaunch: function() { require('ts-node').register({ project: 'e2e' }); }, onPrepare: function() { jasmine.getEnv().addReporter(new SpecReporter()); } };

protractor.sauce.conf.js file

We can notice that there are two environment values: SAUCE_USERNAME and SAUCE_ACCESS_KEY. We can set theses values in our Travis CI account in the settings sections of our project. The information can be found in our Sauce Labs account settings.

Protractor is an end-to-end test framework for Angular. Protractor runs tests against our application running in a real browser, interacting with it as a user would.

e2e configuration

In the e2e folder of our application, we need to place a file called tsconfig.json.

{ "compileOnSave": false, "compilerOptions": { "declaration": false, "emitDecoratorMetadata": true, "experimentalDecorators": true, "module": "commonjs", "moduleResolution": "node", "outDir": "../dist/out-tsc-e2e", "sourceMap": true, "target": "es5", "typeRoots": [ "../node_modules/@types" ] } }

tsconfig.json file in e2e folder

We also need to place a similar file at the root of our application.

{ "compileOnSave": false, "compilerOptions": { "outDir": "./dist/out-tsc", "baseUrl": "src", "sourceMap": true, "declaration": false, "moduleResolution": "node", "emitDecoratorMetadata": true, "experimentalDecorators": true, "target": "es5", "typeRoots": [ "node_modules/@types" ], "lib": [ "es2016", "dom" ] } }

tsconfig.json file

End-to-end (e2e) tests explore the application as users experience it. In e2e testing, one process runs the real application and a second process runs Protractor tests that simulate user behavior and assert that the application respond in the browser as expected.

Coveralls

In our folder, we are now going to create a file called .coverall.yml with the token corresponding to our respository that can be found in our Coveralls account.

repo_token: YOUR_TOKEN

.coverall.yml file

Travis CI

Now it is time to tell Travis CI what to do with our files. Let's create a file called .travis.yml and fill it like so:

language: node_js sudo: true dist: trusty node_js: - '6' branches: only: - master env: global: - CHROME_BIN=/usr/bin/google-chrome - DISPLAY=:99.0 cache: directories: - node_modules before_install: - ./scripts/install-dependencies.sh - ./scripts/setup-github-access.sh after_success: - ./scripts/delete-gh-pages.sh - git status - npm run build-gh-pages - npm run deploy-gh-pages - git checkout master - sleep 10 - tsc -p e2e - npm run e2e ./config/protractor.sauce.conf.js notifications: email: false

.travis.yml file

GitHub Token

Before we go any further, we need to create an access token on our GitHub account. We can do that in the settings of our account.

Scripts

In the previous section, we told Travis CI to use three bash scripts: install-dependencies.sh, setup-github-access.sh and delete-gh-pages.sh. We are now going to create a folder called scripts and create those three different scripts like so:

The first script, as its name lets us figure out, just install our dependencies.

#!/bin/bash export CHROME_BIN=/usr/bin/google-chrome export DISPLAY=:99.0 #Install chrome stable version sh -e /etc/init.d/xvfb start sudo apt-get update sudo apt-get install -y libappindicator1 fonts-liberation wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb sudo dpkg -i google-chrome*.deb rm -f google-chrome-stable_current_amd64.deb

install-dependencies.sh file

The second script ensures that we can access to our GitHub repository during the build. We can see in this script that a variable called $GH_TOKEN is used. This environment variable can be set in Travis CI by clicking on Settings in our repositories list.

#!/bin/bash set -e echo "machine github.com" >> ~/.netrc echo "login [email protected]" >> ~/.netrc echo "password $GITHUB_TOKEN" >> ~/.netrc

setup-github-access.sh file

The last script deletes the branch gh-pages to let us deploy our app on GitHub Pages (we can't do it if the branch is already here).

#!/bin/bash set -e git ls-remote --heads | grep gh-pages > /dev/null if [ "$?" == "0" ]; then git push origin --delete gh-pages fi

delete-gh-pages.sh file

Now, we need to tell GitHub and Travis CI that these files are executable. We can do that with the following command:

chmod +x the_file

Git command to change chmod

Perhaps the following command may also be needed:

git update-index --chmod=+x the_file

Git command to update index

package.json

We now have to make a few adjustments in our package.json file, more specifically, in the scripts section. We can see that use once again the $GH_TOKEN variable.

"scripts": { "ng": "ng", "start": "ng serve", "build": "ng build", "test": "ng test --code-coverage true --watch false && cat ./coverage/*/lcov.info | ./node_modules/coveralls/bin/coveralls.js", "lint": "ng lint", "pree2e": "webdriver-manager update --standalone false --gecko false", "e2e": "protractor", "build-gh-pages": "ng build --prod --base-href \"/YOUR_REPOSITORY/\"", "deploy-gh-pages": "angular-cli-ghpages --repo=https://[email protected]/YOUR_USERNAME/YOUR_REPOSITORY.git --name=YOUR_USERNAME --email=YOUR_EMAIL" }

package.json file

app.po.ts

We need to make one last adjustment in the file called app.po.ts that we can find in the e2e folder. Let's make it like so:

import { browser, element, by } from 'protractor'; export class BookreaderPage { navigateTo() { browser.ignoreSynchronization = true; return browser.get(browser.baseUrl); } getParagraphText() { return element(by.css('app-root h1')).getText(); } }

app.po.ts file

Here we go!

Now, we can push (again) our files and, if everything is alright, the magic will happen! We just have to check Travis CI, Coveralls and Sauce Labs.

[mlbors]

First steps with continuous integration

In this post, we are going to overview some tools to use continuous integration.

Continuous integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early. CI originated from within the extreme programming paradigm, but the principles can be applied to any iterative programming model, such as agile programming.

There are many tools that we can use and one of the most famous is probably Jenkins, written in Java. But we are not going to talk about Jenkins in this post. Instead, we are going to generate a really small project with some other tools. For that, we need Git, Ruby, PHP, Composer, PHPUnit, PHP-CS-Fixer, GitHub, Travis, StyleCI and Codecov. We may think that is a lot of stuff, but there is no reason to be afraid.

Definitions

First of all, a few and short definitions!

Git

Git is a free and open source distributed version software for tracking changes in computer files and coordinating work on those files among multiple people. It was created by Linus Torvalds.

GitHub

GitHub is a web-based Git that got a mascot called Octocat, a cat with five tentacles and a human-like face. It was created by Chris Wanstrath, PJ Hyett and Tom Preston-Werner.

Ruby

A famous programming language. We will need it to install gems (Ruby Libraries).

PHP

Another famous programming language. We will create our project with it.

Composer

A dependency manager for PHP that was initially developed by Nils Adermann and Jordi Boggiano.

PHPUnit

A framework for unit testing for PHP created by Sebastian Bergmann.

PHP-CS-Fixer

A tool to automatically fix PHP coding standards issues.

Travis CI

A continuous integration service used to build and test software projects hosted at GitHub.

StyleCI

StyleCI makes sure that all code on our PHP project is written to a consistent standard.

Codecov

Codecov provides metrics and insights into the results of tests through code coverage.

For the sake of this article, we are not going to discuss here about test driven development (TDD), how to write good and efficient tests and things like that. These are other subjects. The aim of this post is to see how to install and use all of those tools.

Installation

Composer

Installing Composer is really easy. We just have to go to https://getcomposer.org/download/ and follow the instructions. On Windows, there is an installer.

PHPUnit

Again, we have to go to https://phpunit.de/manual/current/en/installation.html and follow the instructions. On Windows, globally install PHPUnit is a bit trickier.

Ruby

On a Linux system and on macOS, Ruby is already here. On Windows, we have to download the installer from https://rubyinstaller.org/downloads/.

Git

We just need to go here https://git-scm.com/downloads and follow the instructions.

PHP-CS-Fixer

On a Linux system and on macOS, we can globally install PHP-CS-Fixer by following the instructions written here https://github.com/FriendsOfPHP/PHP-CS-Fixer.

GitHub, Travis CI, StyleCI, Codecov

We just have to go on GitHub (https://github.com/) and create an account. We will then have access to Travis CI (https://travis-ci.org/), StyleCI (https://styleci.io/) and Codecov (https://codecov.io) with that same account.

Travis gem

The gem can be install by doing the following command in the terminal:

gem install travis

Setting up the project

Now that we have all our new tools, we can move to the next step and set up our project.

Composer

The first thing we need to do is to create a composer.json file at the root of our project. We can create it manually or by doing the following command:

composer init

Command

{ "name": "mlbors/simple-travis-codevoc-styleci", "description": "Simple project using Composer, PHPUnit, Travis, Codecov, PHP-CS-Fixer and StyleCI", "license": "MIT", "require": { "php": ">=5.6", "ext-curl": "*", "friendsofphp/php-cs-fixer": "2.0.0" }, "require-dev": { "phpunit/phpunit": "5.7.*", "vlucas/phpdotenv": "^2.0" }, "autoload": { "psr-4": { "mlbors\\SimpleApp\\": "app/" } }, "autoload-dev": { "psr-4": { "mlbors\\SimpleApp\\Tests\\": "tests/" } }, "homepage": "https://github.com/mlbors/simple-travis-codevoc-styleci", "authors": [ { "name": "mlbors", "email": "[email protected]", "homepage": "http://www.mlbors.com/" } ] }

composer.json file

In the above example, in the require section, we say that we need PHP 5.6 or higher, Curl and PHP-CS-Fixer. In require-dev, we have PHPUnit and PHPDotEnv. In the autoload and autoload-dev sections, we say that we use the PSR-4 standard and how our namespaces will be.

It is time to do the following commands to install and generate the autoload:

composer install composer dump‑autoload -o

GitHub

First of all, we create two files: .gitignore and .gitattributes.

/vendor /node_modules .env .php_cs.cache composer.lock npm-debug.log

.gitignore file

/.gitattributes export-ignore /.gitignore export-ignore /phpunit.xml export-ignore /.env export-ignore /.env.example export-ignore

.gitattributes file

Then, we go on GitHub and create a new repository. Then, on our system, at the root of our project, we do the following commands:

git init git add . git commit –m "First commit" git remote add origin https://github.com/user/repository.git git push –u origin master

Replace user and repository by your username and your repository

Now we can go on Travis CI, StyleCI and Codecov and link our repository to those services.

StyleCI

Now, at the root of our project, we create the styleci.yml that will tell StyleCI file which rules we use.

preset: psr1

.styleci.yml file

.env File

The .env file allows us to place some values that we do not want to appear in our code. In our example, we can load them with PHPDotEnv. The .env file will be ignored by Git but we can create a .envexample file with fake values just to tell the world that we need a .env file.

SIMPLE_KEY="someValue"

.env file

Travis CI

Now we can create the .travis file that will tell Travis CI what to do with our files. We can also include critical values in that file, for example, the ones that are in the .env file, by encrypting them like so:

travis encrypt SOMEVAR=secretvalue

Command

language: php php: - 5.6 - 7.0 install: - sudo pip install codecov before_script: - composer self-update - composer install --no-interaction after_success: - bash <(curl -s https://codecov.io/bash) script: - vendor/bin/phpunit --coverage-clover=coverage.xml env: - SIMPLE_KEY=someValue notifications: email: false

.travis file

In the above example, we tell Travis CI a few things: we state that we use PHP and our app will have to work on PHP 5.6 and PHP 7. We also install the Python codecov package with pip for code coverage. Our dependencies will also be installed with Composer and PHPUnit will be executed. If everything is alright, we send the information to Codecov.

More information about how to customize the build are available here!

PHP-CS-Fixer

The .php_cs file tells PHP-CS-Fixer the rules that we use.

notPath('vendor') ->in(__DIR__) ->name('*.php') ->ignoreDotFiles(true) ->ignoreVCS(true); $rules = [ '@Symfony' => true ]; return PhpCsFixer\Config::create() ->setRules($rules) ->setFinder($finder) ->setUsingCache(false);

.php_cs file

PHPUnit

We create a phpunit.xml to tell PHPUnit how to run our unit tests. We also set the processUncoveredFilesFromWhitelist attribute for code coverage.

<?xml version="1.0" encoding="UTF-8"?> <phpunit backupGlobals="false" backupStaticAttributes="false" bootstrap="vendor/autoload.php" colors="true" convertErrorsToExceptions="true" convertNoticesToExceptions="true" convertWarningsToExceptions="true" processIsolation="false" stopOnFailure="false" syntaxCheck="false"> <testsuites> <testsuite name="SimpleApp Test Suite"> <directory suffix="Test.php">./tests/</directory> </testsuite> </testsuites> <filter> <whitelist processUncoveredFilesFromWhitelist="true"> <directory suffix=".php">./app</directory> </whitelist> </filter> </phpunit>

.phpunit.xml

In the above example, among other things, we tell PHPUnit where our test suite will be and how to compute the code coverage.

README

The README.md file tells the world about our project. We can also include into it the badges that Travis CI, StyleCI and Codecov will generate.

[](https://travis-ci.org/mlbors/simple-travis-codevoc-styleci) [](https://styleci.io/repos/75407002) [](https://codecov.io/gh/mlbors/simple-travis-codevoc-styleci) Simple project using Composer, PHPUnit, Travis, Codecov, PHP-CS-Fixer and StyleCI

.README.md file

Let's use all those tools

PHP-CS-Fixer

To use PHP-CS-Fixer, we can enter the following commands in our terminal:

$ php php-cs-fixer.phar fix /complete/path/to/dir $ php php-cs-fixer.phar fix /complete/path/to/file

On Windows we can do it like so:

php path\to\project\vendor\friendsofphp\php-cs-fixer\php-cs-fixer fix path\to\fileordir

PHPUnit

To run our tests, we just have to use

phpunit

On Windows, we can do it like so:

php path\to\project\vendor\phpunit\phpunit\phpunit

And we push

It is time! Now we can do

git add . git commit –m "This is a commit" git push origin mybranch

If everything is alright, our files will be send to GitHub, then to Travis CI and StyleCI. Travis CI will do what we wrote in our .travis file then send coverage information to Codecov. And voilà!

Advantages

Continuous integration advantages are several:

Integration bugs are detected early and are easy to track down

Constant availability of a "current" build for testing, demo, or release purposes

Frequent code check-in pushes developers to create modular, less complex code

Enforces discipline of frequent automated testing

Immediate feedback on system-wide impact of local changes

And so on. On the other hand, we can state that continuous integration requires a considerable amount of work and time. And 100% code coverage doesn't mean that every execution path is tested and our software is bug free.

You can find the complete example on my GitHub (https://github.com/mlbors/simple-travis-codevoc-styleci).

[luca-tumedei]

Travis CI, PHP 5.2 and PHPUnit

Travis CI, PHP 5.2 and PHPUnit

Testing a PHP 5.2 compatible package on Travis with PHPUnit. Why bother? In a canned answer: due to WordPress minimum requirements. While PHP 5.2 has been deprecated for some time there is still software that will probably not run on it (a little portion of WordPress installations still runs on PHP 5.2). Second best answer for the thrill of it. Third answer: there are some edge cases that will…

View On WordPress

[luca-tumedei]

Using Phantoman on Travis CI

Using Phantoman on Travis CI

Using Travis CI built-in phantomjs along Phantoman Codeception extension. Enter the phantom PhantomJS is a steady presence in any acceptance test that will require JavaScript support. While using Codeception I opt for faster solutions like PhpBrowser or the WPBrowser module for wp-browser but JavaScript might be integral to the application and in those cases a JavaScript-capable web driver comes…

View On WordPress

[for-the-user]

my dorky continuous delivery solution with git and travis ci

I'm actually pretty excited about this, even though I know everyone is using containers nowadays. I'll get up to that soon!

So my stack is:

flask gunicorn supervisor nginx

The workflow or pipeline is: 1. Write some code locally 2. Run unit tests locally 3. Check the app in a browser locally 4. Commit your changes to git 5. Push your changes 6. Travis ci will check out your changes and run the unit tests 7. Travis ci will send a webhook notification to the app 8. The app will check the POST data from travis ci 9. If the message contians success, the app will execute a bash script 10. The bash script will do a git pull 11. The bash script will send a HUP signal to gunicorn processes 12. Supervisor will see that gunicorn process have hung up 13. Supervisor will spawn new ones containing the new code which has just been pulled from git, with no outage.

From git pushing to a successful deployment, it all takes about 1 minute, and when you reload your staging or prod app, it should have the changes, as long as travis builds successfuly.

So these are the 'extra' bits in my python web application that are required for all of this to happen.

The folder structure looks something like this:

├── deploy.sh ├── myapp.wsgi ├── .git ├── .gitignore ├── index.py ├── README.md ├── requirements.txt ├── static │   ├── index-style.css ├── templates │   └── index.html ├── tests.py └── .travis.yml

myapp.wsgi, index.py, static/, templates/ are the basic app. The .git folder and .gitignore folders are required for git. tests.py contains unit tests. .travis.yml is required to tell travis ci what to do on a new commit. requirements.txt contains build dependancies which travis ci will use. deploy.sh is a bash script containing two commands.

The contents of the .travis.yml:

sudo: required language: python python: - '2.7' compiler: - gcc install: - pip install -r requirements.txt script: - python tests.py notifications: webhooks: urls: - https://www.myapp.com/random_string on_success: always on_failure: never

This instructs travis ci what environment to set up for your app, what dependencies need installing, where your tests are and how to notify you when the build has finished. With the webhooks instruction, travis will send a http POST to the url you specify if the build was successful. Travis build notification documentation is here. You can read about the format and content of the POST there.

In the app itself, you need to create an endpoint or route that matches the webhook url you've told travis to POST to after a successful build. So in index.py, add a new view and some extra libraries:

import json, urllib from hashlib import sha256 from subprocess import call . . . @app.route('/random_string', methods=['POST']) def random_string(): response = {} if request.method == 'POST': if request.headers['Authorization'] == sha256('my_git_username/my_git_repo' + my_travis_auth_token).hexdigest(): response = request.get_data() response = urllib.unquote(response).decode('utf8') response = response.replace("payload=", "") response = json.loads(response) if response['result'] == 0: if response['result_message'] == 'Passed' or 'Fixed': call("./deploy.sh") resp = Response('{"Status":"Deployed"}', status=200, mimetype='application/json') return resp else: logging.error('Build Error: CI build result message was not successful.', extra={'Build Result Message': response['result_message']}) resp = Response('{"Status":"Broken"}', status=500, mimetype='application/json') return resp else: logging.error('Build Error: CI build result was not successful.', extra={'Build Result': response['result']}) resp = Response('{"Status":"Broken"}', status=500, mimetype='application/json') return resp else: logging.error('Auth Error: Webhook POST request was not from Travis CI.', extra={'Auth Header': request.headers['Authorization']}) resp = Response('{"Status":"Broken"}', status=500, mimetype='application/json') return resp else: logging.error('HTTP Status Error: Method not allowed.', extra={'HTTP Method': request.method}) resp = Response('{"Status":"Broken"}', status=500, mimetype='application/json') return resp

This view will only accept POST requests, if the request headers include some Authorization string from travis, then load the data and look for the result value and message. If the value is what you expect, then run the bash script. Otherwise log some errors and return an error response.

The bash script contains:

#!/bin/bash git pull ps -ef | grep [g]unicorn | awk '{ print $2 }' | xargs kill -HUP

This will do a git pull to the current directory, then it will look for current running gunicorn processes, and then send a hangup signal to each of them.

When the script is triggered to run, this is what gunicorn's logs report is happening:

[2016-08-22 01:33:30 +0000] [3701] [INFO] Listening at: http://0.0.0.0:5000 (3701) [2016-08-22 01:33:30 +0000] [3701] [INFO] Using worker: eventlet [2016-08-22 01:33:30 +0000] [3706] [INFO] Booting worker with pid: 3706 [2016-08-22 01:33:32 +0000] [3701] [INFO] Handling signal: hup [2016-08-22 01:33:32 +0000] [3701] [INFO] Hang up: Master [2016-08-22 01:33:32 +0000] [3717] [INFO] Booting worker with pid: 3717

3701 is the master process, and 3706 was a child. We hung up 3706 and then a new one was spawned immediately after, 3717. What's happening here is supervisor is watching these processes and ensuring that the running processes match it's configuration. When a process dies or hangs up for whatever reason, it just spawns a new one. You can see the processes being reloaded:

Before:

[root@localhost fromearth]# ps -ef | grep gunicorn root 3701 3693 0 01:33 pts/0 00:00:01 /usr/bin/python /usr/bin/gunicorn -w 1 -k eventlet -b 0.0.0.0:5000 index:app root 3706 3701 0 01:34 pts/0 00:00:01 /usr/bin/python /usr/bin/gunicorn -w 1 -k eventlet -b 0.0.0.0:5000 index:app

After:

[root@localhost fromearth]# ps -ef | grep gunicorn root 3701 3693 0 01:33 pts/0 00:00:01 /usr/bin/python /usr/bin/gunicorn -w 1 -k eventlet -b 0.0.0.0:5000 index:app root 3717 3701 0 01:34 pts/0 00:00:01 /usr/bin/python /usr/bin/gunicorn -w 1 -k eventlet -b 0.0.0.0:5000 index:app

You can read about how to setup supervisor with gunicorn here. Reload your browser and your staging or prod app will have all the new changes. And that is how my dorky solution works. Next time, containers!